OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 3:38 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: File I/O from Kernel
PostPosted: Thu Apr 20, 2017 5:16 pm 
Offline

Joined: Thu Apr 20, 2017 4:48 pm
Posts: 4
I know there are lots of questions like this , but the answers were of no help . I followed the instructions from the Bare Bones - OSDev Wiki http://wiki.osdev.org/Bare_Bones (create a bootloader and load a kernel to output a string) .Everything went fine . But I thought i could create a simple log file for a very basic game .And on google all the answers used some external libraries (<linux/kernel.h> , <fcntl.h> ,<sys/types.h>... etc.) which I don't have or don't know how to link to the kernel.

Notice : A lot of people said that it's not recommended/possible (due to some policy reasons) doing I/O calls from Kernel but also a lot of people say that it is possible and that there's no problem and show some code that isn't C compatible.

I am pretty stuck here .Any suggestion is of help . Thank you .


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Thu Apr 20, 2017 5:47 pm 
Offline
Member
Member

Joined: Mon Jul 25, 2016 6:54 pm
Posts: 223
Location: Adelaide, Australia
The thing is, when you first start writing a kernel, there are no files. You first need some sort of disk driver, to read and write to whatever storage device you're using. That will give you chucks of binary data. Then you need a file system driver, which allows you to interpret the raw binary into files and directories. Then you will have commands equivalent to read and write from Linux.
If you're still in real mode, you can use int 13H instead of writing a native disk driver, but the file system driver you'll have to do yourself.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Thu Apr 20, 2017 6:04 pm 
Offline
Member
Member

Joined: Wed Oct 30, 2013 1:57 pm
Posts: 306
Location: Germany
The thing with operating system development is that you start with absolutely nothing. You have to write everything. It means "[...] that there is no C standard library, only what we provide ourselves." If you want any sort of function that does I/O, you have to write it yourself. End of story.

If you want more specific help you'll have to actually ask a question. For that, you'll need to provide more information. What type of I/O do you want? What did you check out? What exactly are you trying to do?

Also, please tell me, is <linux/kernel.h> a library?

_________________
managarm


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Thu Apr 20, 2017 6:34 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

ReznicencuBogdan wrote:
I know there are lots of questions like this , but the answers were of no help . I followed the instructions from the Bare Bones - OSDev Wiki http://wiki.osdev.org/Bare_Bones (create a bootloader and load a kernel to output a string) .Everything went fine . But I thought i could create a simple log file for a very basic game .And on google all the answers used some external libraries (<linux/kernel.h> , <fcntl.h> ,<sys/types.h>... etc.) which I don't have or don't know how to link to the kernel.


When you read a file there are multiple steps at multiple layers. For the highest level:
  • VFS does a recursive meta-data walk (e.g. for "/foo/bar/baz.txt" it checks for a directory called "/foo", then checks for the sub-directory "bar", then checks for "baz.txt"). This requires some kind of cache (otherwise it's a massive performance disaster), where VFS to ask a file system to fetch directory info when there's a "VFS meta-data cache miss". At each step during this recursive meta-data walk VFS is checking the types (e.g. is "bar" a symbolic link or a mount point) and doing file permission checks.
  • If the recursive meta-data walk was fine; VFS uses the entry (in its meta-data cache) to find out if the file's data is in the VFS file data cache. VFS asks a file system to fetch the file data when there's a "VFS file data cache miss".
Note that managing these caches efficiently means doing things like pre-fetching data from disk into VFS cache (when there's lots of free RAM and the hardware (disk controllers, etc) have nothing better to do; and allowing the RAM to be taken back by the kernel when it needs more free RAM. Also all of this should be asynchronous - if one process asks the VFS for something that results in any kind of "VFS cache miss" then you don't want all other processes to have to wait for that, especially when other processes are trying to do things that can be satisified from VFS cache. Ideally, even when everything is a "VFS cache miss" you still want "many file IO requests in flight" so that (e.g.) storage device drivers can optimise disk access patterns (minimise seeks, etc), and "IO priorities" so that the lower layers (file system and storage device drivers) have some clue about what is/isn't important (and can make sure more important things happen sooner).

For the next level (file systems); in general, you get a request to fetch something from VFS (directory info, or file data) and convert them into requests to fetch something from a storage device driver (blocks/sectors from a partition). This depends on what kind of file system, and it involves caching things that VFS doesn't already cache (e.g. the "cluster allocation table" for FAT). Of course this also needs to be asynchronous, and should keep track of "pending requests" and perform them in an optimised order (and not necessarily the order requests arrive) taking into account the IO priorities.

For the next level (storage device drivers); in general, you get a request to fetch something from a file system (or kernel for swap space) and convert them into requests for the hardware itself to do. This also needs to be asynchronous, and should keep track of "pending requests" and perform them in an optimised order (and not necessarily the order requests arrive) taking into account the IO priorities.

Now....

Before you can write a storage device driver, you will need:
  • Support for physical and virtual memory management
  • Support for PCI bus enumeration
  • Support for starting drivers
  • Support for IRQ handling
  • Support for "memory mapped PCI" areas
  • Support for time (e.g. "nanodelay()", time-outs, etc)
  • Ideally; a documented "storage device driver interface" design


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 1:24 am 
Offline
Member
Member
User avatar

Joined: Thu Mar 10, 2016 7:35 am
Posts: 167
Location: Lancaster, England, Disunited Kingdom
How to persuade someone to give up in one long post!!!

What Brendan describes is the ultimate end after some years experience and improvements.
What you actually need first is a routine to read or write a group of contiguous sectors to and from a disc unit somewhere on your system that you have dedicated to your OS. I have a partition on a hard disc not used by any other OS.

That's the first target. Once you can do that you can read in, edit and write out. Magic. You'll feel great. It doesn't need any of Brendan's optimisations (yet). It doesn't need to have queues and it can be as synchronous as you like. There's only you using it and you're only doing one thing at once at the moment and you're not using a lot of data. It's fine. Keep it simple. It will be fast enough for your needs.

A file system would be nice of course. A first stab that'll cope with a dozen different files is easily set up in five minutes with a sheet of A4 paper and a ballpoint pen on which you have three columns:
File Name. Start Cluster. Finish cluster
Make each file a lot bigger than it needs to be: delays the time when you will need to decide what to do when you want to resave a file, but it has grown beyond the space allocated to it.

Now the first of these dozen files (or even the seventh if you want to be unconventional) could be used as a file index to replace your sheet of A4 paper which is getting a bit scruffy and has a lot of crossings out and alterations. Keep this first file (if you want you can rename is as a file index rather than a file if that makes you feel more professional, but in reality it's perfectly OK being just a file) - now what was I saying - keep this first file simple. You don't NEED directories (yet). You can now write code to consult this file for a filename and use the information to read it in and write it back out again. Then you can add a routine to create a new file and to rename a file. And put a listing on screen.. And...

The secret is that each stage is just a gentle improvement on the previous one. And don't be worried that what you do today won't be compatible with what you need tomorrow. Tomorrow you'll have a lot more experience and will feel confident about throwing your juvenile code away and doing everything a lot better second time round.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 2:17 am 
Offline
Member
Member
User avatar

Joined: Thu Jul 12, 2012 7:29 am
Posts: 723
Location: Tallinn, Estonia
Long story short: if you have a game in your kernel and it needs to write a log file you're doing it all WRONG.

It's not an OS, it's a bare-metal game and have little to do with OSdev.

_________________
Learn to read.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 3:45 am 
Offline
Member
Member
User avatar

Joined: Fri Oct 03, 2008 4:13 am
Posts: 153
Location: Ogre, Latvia, EU
Question to OP: do you really need to write a log file? Or just a simple way to write log?

I would suggest to send your log via serial port to another computer and view/store it in more mature OS (Linux, Windows). And if you run your OS/game in emulator, you can capture serial output as well. Driver for serial port is pretty simple to make.

_________________
If something looks overcomplicated, most likely it is.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 5:49 am 
Offline

Joined: Thu Apr 20, 2017 4:48 pm
Posts: 4
I apreciate the sugestions . But i don't thing i want to get into the file system for the moment.Isn't there some easy way to write some chunks of data (some gibberish for the moment like some system color(store default color),system properties(some other silly thing)) on some empty sector on the harddisk ? It might be a silly question , but I am not really that deep with the info.Still have a lot to learn.

Quote:
What you actually need first is a routine to read or write a group of contiguous sectors to and from a disc unit somewhere on your system that you have dedicated to your OS. I have a partition on a hard disc not used by any other OS.
Ok . How can i achieve this ?
Can you please provide me some references , online documents?Searched but couldn't find something usefull.

Thank you.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 6:23 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
I don't think you have searched hard enough. There is loads of information about this on this wiki and the wider Internet.

I don't think people are going to do your research for you.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 7:15 am 
Offline

Joined: Thu Apr 20, 2017 4:48 pm
Posts: 4
iansjack wrote:
I don't think you have searched hard enough. There is loads of information about this on this wiki and the wider Internet.

I don't think people are going to do your research for you.


Every single person on this forum did the same thing .Said "there is lots of information about this .... you didn't search enough." and that was all. I didn't say there isn't . I said i found but couldn't implement , and so I asked for advice on this topic (some low level). So all you did was to fill up this page for nothing .Your intervention in this case was useless. You spent some minutes of your life for nothing . I asked for some consistent support , not useless activity.

If you were to be In my place , how would you feel when someone like you would post an answer like yours after having spent hours looking for an answer , a good one?


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 8:20 am 
Offline
Member
Member
User avatar

Joined: Thu Aug 06, 2015 6:41 am
Posts: 97
Location: Netherlands
ReznicencuBogdan wrote:
I apreciate the sugestions . But i don't thing i want to get into the file system for the moment.Isn't there some easy way to write some chunks of data (some gibberish for the moment like some system color(store default color),system properties(some other silly thing)) on some empty sector on the harddisk ? It might be a silly question , but I am not really that deep with the info.Still have a lot to learn.

Quote:
What you actually need first is a routine to read or write a group of contiguous sectors to and from a disc unit somewhere on your system that you have dedicated to your OS. I have a partition on a hard disc not used by any other OS.
Ok . How can i achieve this ?
Can you please provide me some references , online documents?Searched but couldn't find something useful.

You need to learn how to search.

First of all, let me give you my definition of useful information: Anything that either helps to understand how to accomplish my goal (r/w to hard disk, easy-mode), or points me to information that does. This includes anything that might contain useful links (the internet is full of them!) or even keywords I can throw at google that I didn't know before.

With that in mind, the general steps I use when looking (OSDev related) things up is:
  • Look at the wiki.osdev.org main page to see if I can find anything that might be useful information
  • If that doesn't work, search the wiki using its search bar. Use any keywords you know, preferably as specific as possible, and preferably something that sounds likely to be related to OSDev or hardware programming (e.g. don't type 'screen' if you think the thing you're looking for might be called 'int 10h', 'VGA', 'VESA', or 'VBE')
  • If that doesn't work, try searching the forum or google using the same method

Using this strategy, pretending I didn't know anything, I would have found the following things:
  • A box on the main wiki page called 'hardware' (harddisks are hardware, so if it's anywhere on this page, it'll be here)
  • A list of links in this box titled 'storage devices' (so far so good, getting warmer)
  • A link called 'ATA (harddisks)' (idk at this point what ATA is, it might be edible for all I know, but '(harddisks)' sounds good so I'll click on it)
  • A bloody gold mine! Do you see all those links? I don't know where to start... But I'll certainly remember all those words because at least one of them must be the golden keyword that will unlock all the information I need on harddisks, should I need to search elsewhere.
  • But wait, there's a list of links above which it says: 'To get started quickly, try one of the following pages', that's perfect, because I want to get started quickly, and pages that tell me how to do so are exactly what I'm looking for!
    • Something about writing drivers, sounds fantastic, also sounds complicated, I'll click on it if nothing easier catches my eye...
    • Something with ATA Packet Interface, SCSI, commands, protocol, CD-ROM, something else needed to get started... sounds useful but certainly not the basics I'm looking for...
    • ATA in realmode (BIOS)... (I'm not sure if you're into realmode or pmode, so I'll assume you'll use whichever one is easiest), it says 'If you want to make disk accesses using the legacy BIOS', well I certainly want to make disk accesses, and I don't know what this BIOS thing is, but it says you can use it to access disk, so it might be a library or something similar that can do the work for me, I'll click on that.
  • According to the first three sentences the BIOS is a thing that is still operational (so I don't have to start it? great!) in realmode, provides disk access routines (so I don't have to make my own? great!), better yet, it is the only way to access disk in a bootloader (so it can't be complicated, and might even be easy! After all, the code to use it must fit in a tiny bootloader!) before loading drivers (which means you obviously don't need drivers to use it, amazing!). I'll read this page and use it to write my simple 'r/w to a sector on harddisk' code. The page even contains a piece of example code that I won't copy but will certainly compare with my own code in case of bugs.
  • It seems like int 0x13 is the way to go for accessing disk in realmode, this sounds like a really good keyword in case I need to search for additional information...
  • The way the page talks about (the int 0x13 family of) BIOS calls makes me think there are more than just those for accessing disk, I will be sure to remember to try and put BIOS after any future keywords when looking things up, because who knows what other useful things this BIOS could do for me...
  • A google search yielded a wikipedia article, and what looks like some sort of manual

Spoiler alert: it doesn't get easier than int 13h

Note that it is fine to do everything easy-mode when you're getting started, but if you want to write a proper OS you should take the advice that others have given both on the wiki and here on the forum: do it the best way, even if that's harder than you'd like because it will pay off (instead of hurting you in the long run, which is what the easy but not best way does).


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 9:37 am 
Offline
Member
Member
User avatar

Joined: Sun Sep 19, 2010 10:05 pm
Posts: 1074
ReznicencuBogdan wrote:
Ok . How can i achieve this ?
Can you please provide me some references , online documents?Searched but couldn't find something usefull.

If you followed the barebones tutorial, then you are using C and are in 32-bit protected mode when your kernel runs. This means that using the BIOS (INT 13h) isn't really an option.

This only leaves the option of sending commands directly to the ATA controller. You can find more information on the wiki here: http://wiki.osdev.org/ATA_read/write_sectors

But most of the examples on the wiki are written in Assembly. If you want a C version, the only one I could find on here is this forum post: viewtopic.php?f=1&p=167798#p167798

See if you can at least write some code to write and read registers on the ATA controller properly, and then we can help you communicate with the hard drive.

_________________
Project: OZone
Source: GitHub
Current Task: LIB/OBJ file support
"The more they overthink the plumbing, the easier it is to stop up the drain." - Montgomery Scott


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Fri Apr 21, 2017 9:46 am 
Offline

Joined: Thu Apr 20, 2017 4:48 pm
Posts: 4
Thank you . I appreciate your help.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Sat Apr 22, 2017 1:52 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
ReznicencuBogdan wrote:
I'm not sure If you're aware of the fact that this is a forum for OS developers.

Thank you for enlightening me as to the purpose of these forums.


Top
 Profile  
 
 Post subject: Re: File I/O from Kernel
PostPosted: Sun Apr 30, 2017 1:10 pm 
Offline
Member
Member

Joined: Fri May 20, 2016 2:29 pm
Posts: 77
Location: Paris, France
sleephacker wrote:
You need to learn how to search.

First of all, let me give you my definition of useful information: Anything that either helps to understand how to accomplish my goal (r/w to hard disk, easy-mode), or points me to information that does. This includes anything that might contain useful links (the internet is full of them!) or even keywords I can throw at google that I didn't know before.

What a great response! I don't have anything to add to OP's question that hasn't already been covered. I just wanted to note that you and MichaelFarthing provided great, relevant, and approachable responses. Thanks to MichaelFarthing's blueprint, I now have an ATA driver (using PIO, unfortunately) and an extremely basic on-disk filesystem. No directories yet, but as he notes, gentle improvements are the name of the game :D Brendan gives great info as always, just out of the scope of what OP was looking for.

This forum is great. Thanks for everything guys

_________________
www.github.com/codyd51/axle.git


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Majestic-12 [Bot] and 54 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group