OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Nov 12, 2019 1:31 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 26 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Tue Aug 13, 2019 11:46 am 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
Octocontrabass wrote:
Ethin wrote:
How would I determine this at runtime? Would I just mask all the bits anyway?

Yes.

Ethin wrote:
I get this information by calling get_vendor_string(), get_device_string(), and get_class_string(), all of which are defined in src/pcidb.rs. Its a rust implementation of the entire PCI ID database.

But get_device_string() tries to identify the device using only the device ID, which means it will be wrong if more than one vendor assigns the same device ID. It needs to work like get_subclass_string(), taking both the vendor ID and the device ID as input.

With the current setup of that file that's going to be pretty hard to do. I have a small Python script that does the generation of that file for me (though its primitive and has some errors). Not really sure how to correlate vendor ID and device ID together. The script, as I wrote it, is below:
Code:
import requests
import sys
db=requests.get("https://pci-ids.ucw.cz/v2.2/pci.ids").content.decode()
lines = db.split("\n")
vendors = {}
devices = {}
# extract vendors
# syntax: vendor  vendor_name (no tab)
for line in lines:
  if line.startswith("#"): # comment
    continue
  if line.startswith("\t"): # device
    continue
  if line.startswith("\t\t"): # subsystem
    continue
  if line.startswith('C'): # PCI class
    continue
  if len(line) > 0:
    vendor = line.split("  ")[0]
    vendor_name = line.split("  ")[1]
    vendors[vendor] = vendor_name

# Extract devices
# syntax:    device  device_name            <-- single tab
for line in lines:
  if line.startswith("#"): # comment
    continue
  if not line.startswith("\t"): # device
    continue
  if line.startswith("\t\t"): # subsystem
    continue
  if line.startswith('C'):
    continue
  if len(line) > 0:
    device = line.split("  ")
    devices[device[0]] = device[1]

# print match statements
print("""pub fn get_vendor_string(vendor: u16)->&str {
match vendor {""")
for vendor, name in vendors.items():
  print (f"0x{vendor} => \"{name}\",")
print("_ => \"Unknown vendor\",")
print("}")
print("}")

print("""pub fn get_device_string(device: u16)->&str {
match device {""")
for device, name in devices.items():
  try:
    print (f"0x{device[1:]} => \"{name}\",")
  except UnicodeEncodeError:
    continue
print("_ => \"Unknown device\",")
print("}")
print("}")

Looking at the PCI database (as of version 2019.06.20), the file is written like:
Code:
0014  Loongson Technology LLC
   7a00  Hyper Transport Bridge Controller
   7a02  APB (Advanced Peripheral Bus) Controller
   7a03  Gigabit Ethernet Controller
   7a04  OTG USB Controller
   7a05  Vivante GPU (Graphics Processing Unit)
   7a06  DC (Display Controller)
   7a07  HDA (High Definition Audio) Controller
   7a08  SATA AHCI Controller
   7a09  PCI-to-PCI Bridge
   7a0b  SPI Controller
   7a0c  LPC Controller
   7a0f  DMA (Direct Memory Access) Controller
   7a14  EHCI USB Controller
   7a15  Vivante GPU (Graphics Processing Unit)
   7a19  PCI-to-PCI Bridge
   7a24  OHCI USB Controller
   7a29  PCI-to-PCI Bridge

So:
Code:
# Syntax:
# vendor  vendor_name
#   device  device_name            <-- single tab
#      subvendor subdevice  subsystem_name   <-- two tabs

I'll play with the script though and figure something out.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Tue Aug 13, 2019 1:51 pm 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3553
Location: Chichester, UK
Do you really need a database of all vendor/device IDs? Surely you're only interested in those devices that you have written handlers for.

Let's face it, new devices appear every day, so your database is never going to be up to date.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Tue Aug 13, 2019 2:20 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
I know. I'll most likely get rid of it soon -- it was just something I put in there for debugging purposes.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Thu Aug 15, 2019 8:03 am 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
OK, so I'm looking at the SATA 3.4 specification right now. I'm confused on exactly what my kernel needs to implement because the SATA page says that there are three layers, which appears to be inaccurate; with rev. 3.4, there are four: the physical layer, link layer, transport layer and application layer. Each layer is discussed in chapters 7, 8, 9, 10, and 13 (though 11 and 12, Device command layer protocol and Host command layer protocol, respectively, may also be important as well). Chapter 8 deals with OOB and PHY power states, which I assume to be an extension of the physical layer. For reference, rev. 3.4 describes each layer as follows (the Phy and application layers appear not to have an "overview section," so I'll skip them):
Quote:
9.1 Link layer overview
The Link layer transmits and receives frames, transmits primitives based on control signals from the Transport layer, and receives primitives from the Phy layer that are converted to control signals to the Transport layer. The Link layer need not be cognizant of the content of frames.
Host and device Link layer state machines are similar, however the device is given precedence if both the host and device request ownership for transmission.
10.1 Transport layer overview
The Transport layer need not be cognizant of how frames are transmitted and received. The Transport layer simply constructs Frame Information Structures (FISes) for transmission and decomposes received FISes. Host and device Transport layer state differ in that the source of the FIS content differs. The Transport layer maintains no context in terms of ATA commands or previous FIS content.
11.1 Device command layer protocol overview
In the following Device command layer protocols, if the host sends COMRESET before the device has completed processing a command layer protocol, then the device shall start processing the COMRESET protocol from the beginning. If the device receives a Register Host to Device FIS with C bit cleared to zero and the SRST bit set to one before the device has completed processing a command layer protocol, then the device shall start processing its software reset protocol from the beginning. SYNC Escape is used by a host or device to bring the link back to a known state, and may be used for vendor specific recovery of error or hang conditions. After a SYNC Escape is performed, a software reset may be necessary prior to issuing the next command to the device.
12.1 FPDMA QUEUED command protocol overview
This high-level state machine describes the behavior of the host for the Native Command Queuing command protocol. The host behavior described by the state machine may be provided by host software or host hardware and the intent of the state machines is not to indicate any particular implementation.
This class includes NCQ commands.

Not really sure how informative those overviews were (they don't seem very informative). Here is the top-level TOC of my copy of the specification:
  1. Revision history
  2. Scope
  3. Normative references
  4. Definitions, abbreviations, and conventions
  5. General overview
  6. Cables and connectors
  7. Phy layer
  8. OOB and Phy power states
  9. Link layer
  10. Transport layer
  11. Device command layer protocol
  12. Host command layer protocol
  13. Application layer
  14. Host adapter register interface
  15. Error handling
  16. Port Multiplier
  17. Port Selector
  1. Sample code for CRC and scrambling (informative)
  2. Command processing overview (informative)
  3. Device emulation of nIEN with interrupt pending (informative)
  4. I/O controller module (informative)
  5. Jitter formulas without SSC (informative)
Should I implement AHCI first, then return back to this? Or attempt to just implement this? According to the PCI ID database, git revision b96c3bdee5c1108b169b9020a04fd6d14861a26a (version 2019.08.08), SATA controllers have a class of 01h, a subclass of 06h, and a prog if (I think, if I'm reading the file right) of either 01h (AHCI 1.0), 02h (Serial Storage Bus), or 00h (Vendor specific).


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Thu Aug 15, 2019 9:32 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
I've decided to go with AHCI (since the SATA spec doesn't appear to describe the SATA MMIO structure, bits, etc., but AHCI does). I have an AHCI driver that, currently, does the following:
1. It locates an AHCI device by checking the class, subclass, and prog if. If it finds one, it continues to the next set of steps. Otherwise it skips and continues down the PCI device list.
2. It locates the first BAR that is memory-based and not I/O based and selects that as the default BAR.
3. It sets bit 31 of the GHC register and writes that to ABAR + GHC.
4. It reads the PI register, then reads the CAP register and begins iterating through the bits in the PI register using CAP.NP as its maximum.
A. If the bit is set, it calculates the port address (ABAR + 0x100 + (port * 0x80).
B. It clears the PXCMD register by writing bits 28 .. 31 and setting them to 1 to bring that port into the active state while writing the rest as zeros.
C. It repeatedly reads the PXCMD register until the state returns to idle and then exits the loop.
Is this a good start? Am I on the right track or doing something wrong? I'm committing my code right now; the driver is in src/drivers/storage/ahci.rs.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Fri Aug 16, 2019 1:06 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 1676
Well, I didn't manage to get a post in before you started, but the AHCI specification is where to start for an AHCI driver. You'll also need to refer to the ATA ACS specification for information about the commands you can send to the drive, and maybe the SATA specification for things that aren't explained in the AHCI or ATA specs.

Ethin wrote:
2. It locates the first BAR that is memory-based and not I/O based and selects that as the default BAR.

According to the AHCI specification, you should always use BAR5 instead of searching.

I'm not familiar enough with AHCI to say if there are any other problems.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Fri Aug 16, 2019 5:13 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
Octocontrabass wrote:
Well, I didn't manage to get a post in before you started, but the AHCI specification is where to start for an AHCI driver. You'll also need to refer to the ATA ACS specification for information about the commands you can send to the drive, and maybe the SATA specification for things that aren't explained in the AHCI or ATA specs.

Ethin wrote:
2. It locates the first BAR that is memory-based and not I/O based and selects that as the default BAR.

According to the AHCI specification, you should always use BAR5 instead of searching.

I'm not familiar enough with AHCI to say if there are any other problems.

Thanks for the specification references and help, I'll acquire those (I already have the SATA IO 3.4 specification). I'm now getting these from my kernel and a bit worried:
Quote:
ahci: Mis-aligned write to addr 0x0000000000000005
ahci: Mis-aligned write to addr 0x0000000000000006
ahci: Mis-aligned write to addr 0x0000000000000007
ahci: Mis-aligned write to addr 0x0000000000000009
ahci: Mis-aligned write to addr 0x000000000000000a
ahci: Mis-aligned write to addr 0x000000000000000b

This is odd, the only reads/writes are:
* ABAR + GHC (set bit 31)
* Reading from ABAR + PI and ABAR + CAP.NP
* Reading from PORTADDR + PXCMD, and then to set bits 28 to 31 to 1 to bring the port power state to active (I think these are the right bits, my PDF reader got a bit messed up with that), now leaving the rest of the PXCMD bits alone. Then writing the new command bits to PORTADDR + PXCMD.
* And, finally, repeated reads of PORTADR + PXCMD, bits 28 to 31, to wait until the port becomes ready.
Those are the only reads/writes I'm doing right now. I'm using two functions that I wrote -- read_memory and write_memory -- to do this because I didn't want duplicated code everywhere. They are as follows:
Code:
// ...
use core::ptr::*;
//...
pub fn read_memory(address: u64) -> u64 {
    let addr: *const u64 = address as *const u64;
    unsafe { read_volatile(addr) }
}

pub fn write_memory(address: u64, value: u64) {
    let addr: *mut u64 = address as *mut u64;
    unsafe {
        write_volatile(addr, value);
    }
}


I'm not really sure why this is miss-aligned...
Edit: would someone mind posting the bits for PXCMD? Clearly I'm either not setting the right bits or something else is wrong. Here's how my PDF reader displays the table -- it certainly is not elegant.
Quote:
3.3.7 Offset 18h: PxCMD – Port x Command and Status
Bit
Type
Reset
Description
Interface Communication Control (ICC):
This field is used to control power
management states of the interface. If the Link layer is currently in the L_IDLE state, or
L_NoCommPower state, writes to this field shall cause the HBA to initiate a transition to
the interface power management state requested. If the Link layer is not currently in
the L_IDLE state or L_NoCommPower state, writes to this field shall have no effect.
Value
Definition
Fh - 9h
8h
Reserved
DevSleep: This shall cause the HBA to assert the DEVSLP signal
associated with the port; the HBA shall ignore the Device Sleep Idle
Timeout value specified by PxDEVSLP.DITO. Software shall only
request DevSleep when the interface is in an idle state (i.e. PxCI is
cleared to 0h and PxSACT are cleared to 0h); if CAP2.SDS is cleared
to ‘0’ or if the interface is not idle at the time the register is written,
then the HBA shall not assert the DEVSLP signal and the interface
remains in its current state. If CAPS.SDS is set to ‘1’, CAP2.DESO is
set to ‘1’, and PxSSTS.IPM is not set to ‘6h’, then the HBA shall not
assert the DEVSLP signal and the interface shall remain in its current
state. Additionally, the HBA shall not assert the DEVSLP signal until
PHYRDY has been achieved (after a previous de-assertion).
7h
Reserved
Fh - 7h
Reserved
6h
Slumber: This shall cause the HBA to request a transition of the
interface to the Slumber state. The SATA device may reject the
request and the interface shall remain in its current state.
5h - 3h
Reserved
2h
Partial:
This shall cause the HBA to request a transition of the
interface to the Partial state.
The SATA device may reject the
request and the interface shall remain in its current state.
1h
Active:
This shall cause the HBA to request a transition of the
interface into the active state.
0h
No-Op / Idle: When software reads this value, it indicates the HBA is
ready to accept a new interface control command, although the
transition to the previously selected state may not yet have occurred.
31:28 RW 0h
RW/
RO
RW/
RO
When system software writes a non-reserved value other than No-Op (0h), the HBA
shall perform the actions described above (for the value written) and update this field
back to Idle (0h).
If software writes to this field to change the state to a state the link is already in (i.e.
interface is in the active state and a request is made to go to the active state), the HBA
shall take no action and return this field to Idle. For all but DevSleep, if the interface is
in a low power state and software wants to transition to a different low power state,
software must first bring the link to active and then initiate the transition to the desired
low power state. If CAPS2.DESO is cleared to ‘0’, transition to DevSleep may occur
from any other interface state. If CAP2.DESO is set to ‘1’, then DevSleep may only be
transitioned to if the interface is in Slumber.

It pretty much goes on like that. The bits for the HBA registers were fine, those displayed properly -- or at least well enough I could figure them out -- but now its messing up. I'll try another reader while I'm at it and see how that goes.
Edit 2: removed my rant on PDFs... :P
Edit 3: the bit stuff is no longer necessary -- I got it sorted out. :)


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Sat Aug 17, 2019 6:58 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
Update: So I got the mis-alignment issues fixed (apparently setting only a particular set of bits and leaving the rest alone and then writing that new value back is mis-aligned...). I now have an internal set of data structures representing HBA memory registers, ports, and FISs, with the kernel setting up those structures as it scans AHCI devices. It now fully brings the device online (I think), if I'm understanding the AHCI spec properly. So now my question becomes -- how do I send FISs with ATA/ACS commands (read sector(s), write sector(s), ...) to the disk? Unlike C, I can't have pointers in the structure that I can just read/write to because its unsafe and I'm trying to minimize the use of unsafe (though I know I'll use it a lot anyway). I can read things from the HBA memory registers, but I'm not fully sure on sending FISs. Despite the fact that the rebasing source is in C, it should be adaptable -- though I won't be throwing pointers around, but instead just writing the values directly and then updating the structure after each write. Its a bit more complicated but I don't mind it.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Tue Aug 20, 2019 1:12 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 1676
Ethin wrote:
Update: So I got the mis-alignment issues fixed (apparently setting only a particular set of bits and leaving the rest alone and then writing that new value back is mis-aligned...).

You should be able to write back a modified value without getting alignment errors. Something else must be wrong. Unfortunately I don't know enough about Rust to track down the code responsible for the write!


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Tue Aug 20, 2019 1:19 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3553
Location: Chichester, UK
Quote:
I can't have pointers in the structure that I can just read/write to because its unsafe and I'm trying to minimize the use of unsafe
Surely any operation that writes directly to the hardware (or reads from it) is going to be unsafe as the compiler cannot know what side effects this may have or what unrelated events can change the values of the hardware registers.


Top
 Profile  
 
 Post subject: Re: Reading from SATA/ATA/... disks (and maybe solving my PC
PostPosted: Fri Aug 23, 2019 6:25 am 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 87
Location: North Dakota, United States
So I couldn't access OSDev for a while, so I just am now able to access it and post updates (I had to use the IP address to get access to it, then I tried visiting the forums and the domain worked).
1. Yeah, I don't know what was misaligned either. I fixed it by just zeroing it and setting the relevant bits:
Code:
                let cap = read_memory(bars[5]);
                let cap2 = read_memory(bars[5] + CAP2);
                let pi = read_memory(bars[5] + PI);
                for port in 0..=read_memory(bars[5] + CAP).get_bits(0..=4) {
                    if pi.get_bit(port as usize) {
                        printkln!("AHCI: detected AHCI port {}; activating", port);
                        let portaddr: u64 = bars[5] + 0x100 + (port * 0x80);
                        let mut command: u64 = 0;
                        command.set_bits(28..=31, 1);
                        if cap.get_bit(26) {
                            command.set_bit(27, true);
                            command.set_bit(26, true);
                        }
                        command.set_bit(25, false);
                        command.set_bit(24, false);
                        if cap2.get_bit(2) {
                            command.set_bit(23, true);
                        }
                        write_memory(portaddr + PXCMD, command);
                        while read_memory(portaddr + PXCMD).get_bits(28..=31) == command {
                            hlt();
                            continue;
                        }
// ...
                    }
                }
// ...

This is what I'm doing to "activate" a port. Now I just need to figure out how to send FIS packets to the HBA...
2. Yes, that is unsafe (reading from and writing to raw memory and using in/out instructions). But not all my code is unsafe (I've done my best to implement safe interfaces, for example, though that's extremely difficult to do). I'm thinking of doing the same for IO-based stuff (i.e. disk access, network access, ...) by using Go's idea of an IO package (or module, in Rusts case) and then making readers, writers and closers.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 26 posts ]  Go to page Previous  1, 2

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group