OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 5:05 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 23 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Reading and Writing HDD
PostPosted: Mon Aug 30, 2021 9:39 am 
Offline

Joined: Wed Aug 25, 2021 12:16 pm
Posts: 7
Hey all,

I am getting ready to want to be able to read and write to a disk. I'm currently reading up on SATA specifications but before I get too deep into it I wanted to poll the community. Can your OS read and write data to a disk? If so how? Do you jump back and forth between real and protected mode to use INT 13h? or did you write a driver for communicating with the hardware? What kinds of data storage do you have support for?

I'm in new territory here so I kind of wanted to get a feel for how other people tackled the same problem before I end up going down any pointless rabbit holes :lol:

_________________
My toy OS BaseDOS 8)


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Mon Aug 30, 2021 10:21 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
Write a driver. It's one of the easiest bits of hardware to control.


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Mon Aug 30, 2021 10:48 am 
Offline
Member
Member

Joined: Sat Mar 10, 2018 10:16 am
Posts: 296
https://wiki.osdev.org/ATA_PIO_Mode

Here is everything you need to know to start writing driver for HDD.

There are two widely used types of communication with HDD. It is IDE and AHCI. So do not waste time in reading SATA spec now. You need it only for FIS types when you will write driver for AHCI.

_________________
https://github.com/VendelinSlezak/BleskOS


Last edited by Klakap on Mon Aug 30, 2021 11:25 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Mon Aug 30, 2021 11:24 am 
Offline
Member
Member

Joined: Sun Aug 23, 2020 4:35 pm
Posts: 148
I would definitely agree with iansjack: writing a driver is the way to go.
Early on, writing an IDE driver may be easier just to get the ability to read disks especially in emulators like QEMU, but I'd actually recommend writing an AHCI driver instead.
It isn't terribly hard (compared to something like USB controllers) and works with pretty much any computer since 2010.
Plus, AFAIK the spec is completely free from Intel.
If you're using QEMU you can add an AHCI controller and a disk like so:
Code:
(qemu command) -device ahci,id=ahci -drive file=DISK_IMAGE.img,id=disk,if=none,format=raw -device ide-hd,drive=disk,bus=ahci.0


thedude3253 wrote:
Do you jump back and forth between real and protected mode to use INT 13h?
The only thing I ever use real mode/BIOS interrupts for at all anymore is VESA/VBE resolution switches. Switching is slow, especially for disk reads/writes which ideally should be somewhat asynchronous from the OS's perspective (but synchronous from the program's perspective). The only reason I have to do this for VESA/VBE is because I don't feel like writing a driver for tons of graphics cards and I really want the user of my OS to be able to switch resolutions without rebooting their computer.

Good luck!

_________________
My OS: TritiumOS
https://github.com/foliagecanine/tritium-os
void warranty(laptop_t laptop) { if (laptop.broken) return laptop; }
I don't get it: Why's the warranty void?


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Mon Aug 30, 2021 11:40 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
foliagecanine wrote:
Plus, AFAIK the spec is completely free from Intel.

Yes, but some structures (i.e. the FIS) or documented in the SATA spec, which costs $75 for SATA 3.4 and 3.5, $25 dollars for older versions.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Mon Aug 30, 2021 12:06 pm 
Offline

Joined: Wed Aug 25, 2021 12:16 pm
Posts: 7
nexos wrote:
foliagecanine wrote:
Plus, AFAIK the spec is completely free from Intel.

Yes, but some structures (i.e. the FIS) or documented in the SATA spec, which costs $75 for SATA 3.4 and 3.5, $25 dollars for older versions.

So I'm ignorant, what is the FIS? In addition, why would they lock specs behind a paywall? As a userspace dev I'm not very familiar with the economics of these kinds of things

Edit: I spent 5 seconds on the AHCI osdev wiki page and figured out what FIS is lol

_________________
My toy OS BaseDOS 8)


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Tue Aug 31, 2021 2:43 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
TBH I doubt there's a justifiable reason for them locking the specs behind paywalls other than the fact that these are usually non-profit companies. I mean, at least they aren't as bad as PCI-SIG: you want a spec from them, you gotta pay $3,000.00-$5,000.00 -- and I don't even think that what you get is a digital copy.


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Tue Aug 31, 2021 2:57 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Luckily, there are comprehensive PCI books out there. Look at the company Mindshare on Amazon, and they have rather expensive books about PCI and PCIe. But 40 dollars is a lot better then 3000 dollars!

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 5:05 am 
Offline
Member
Member

Joined: Wed Oct 01, 2008 1:55 pm
Posts: 3192
Before you write a disc driver you need to figure out how it should interface with the OS. You need to figure out how you handle caching of disc sectors.

My old disc drivers passed a buffer of sectors descriptors that contained sector # and the linear buffer. My new disc drivers pass a start sector, sector count, and an array of physical addresses to use as buffers.

Modern hardware (AHCI, USB) uses physical addresses in schedules as sources & destinations in DMA operations, and so there is no reason to pass linear addresses to those.

Using the BIOS int 13 interface never was a good idea, and it certainly isn't today when EFI is part of many computers.


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 9:58 am 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
rdos wrote:
Before you write a disc driver you need to figure out how it should interface with the OS. You need to figure out how you handle caching of disc sectors.

My old disc drivers passed a buffer of sectors descriptors that contained sector # and the linear buffer. My new disc drivers pass a start sector, sector count, and an array of physical addresses to use as buffers.

Modern hardware (AHCI, USB) uses physical addresses in schedules as sources & destinations in DMA operations, and so there is no reason to pass linear addresses to those.

Using the BIOS int 13 interface never was a good idea, and it certainly isn't today when EFI is part of many computers.

I mean, not really. Don't over-complicate the objective. If you just want to read and write data (and for writing you have the VWC if supported) you needn't worry about caching of anything.


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 10:05 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Ethin wrote:
you needn't worry about caching of anything.

For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, caching is very important. Think of it as the TLB. A block buffer cache is the disk equivalent of the TLB. Without it, memory access would always have to access memory, and things would be quite slow. Same thing for disk caches. Instead of reading / writing to disk every time, instead you only write when necessary. Without a disk cache, the system would be unacceptably slow.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 10:59 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
And it's particularly important to cache meta-data.


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 12:43 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1593
nexos wrote:
For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, [...] Without a disk cache, the system would be unacceptably slow.
iansjack wrote:
And it's particularly important to cache meta-data.
I agree with both of you in part and disagree in part. Yes, caching is important, however, the engineering principle to remember here is to first make it work, then make it work fast. "The design process is an iterative one." Let the OP first figure out how to actually transfer the data, then stick a cache in there. Whether that takes the form of a block cache or an FS-level metadata cache is secondary. In this case, I would focus my work after the raw data transfer into the parsing of partition tables, then file systems. And then, when that works, then I would look at caching.

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 1:23 pm 
Offline
Member
Member

Joined: Sun Jun 23, 2019 5:36 pm
Posts: 618
Location: North Dakota, United States
nullplan wrote:
nexos wrote:
For SSDs, you're right, block caching is unnecessary as an SSD is very fast. But for hard disks or floppies, [...] Without a disk cache, the system would be unacceptably slow.
iansjack wrote:
And it's particularly important to cache meta-data.
I agree with both of you in part and disagree in part. Yes, caching is important, however, the engineering principle to remember here is to first make it work, then make it work fast. "The design process is an iterative one." Let the OP first figure out how to actually transfer the data, then stick a cache in there. Whether that takes the form of a block cache or an FS-level metadata cache is secondary. In this case, I would focus my work after the raw data transfer into the parsing of partition tables, then file systems. And then, when that works, then I would look at caching.

This is exactly what I was trying to say. Don't implement a cache when your just trying to read/write to disk. Once you've got an FS and partition layer in place, then worry about caching.


Top
 Profile  
 
 Post subject: Re: Reading and Writing HDD
PostPosted: Thu Sep 02, 2021 1:59 pm 
Offline
Member
Member

Joined: Wed Oct 01, 2008 1:55 pm
Posts: 3192
Caching should not be part of the disc driver. It should be in a separate layer. However, it will determine how the disc is accessed. Generally, you should try to read / write as many sectors as possible in the same request, and avoid copying of data. The usual disc interface typically requires copying or using scatter-gather, which not all devices support. Therefore, it's more efficient to send physical addresses and make sure they have proper alignment. Something you get for free with a decent cache function.

IOW, if you do a lot of disc drivers like int 0x13, passing a buffer without alignment requirements, then you miht have trouble adapting this to a interface with better performance. So, you end up having to rewrite the disc drivers once more because you didn't plan properly.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 23 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 72 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group