OSDev.org

The Place to Start for Operating System Developers
It is currently Mon Dec 09, 2019 5:30 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: What generates operating system files/folders, and when?
PostPosted: Tue Oct 29, 2019 9:31 am 
Offline

Joined: Fri Jan 26, 2018 11:43 am
Posts: 24
This is probably a really simple question, but I'm just beginning to think about hard drive access in my operating system and then realised that I actually don't know what bit of software creates the OS filesystem (for example /bin, /etc, etc.. on linux), and when it's generated.

Does the kernel, when booting up, check that all of these folders/files exist, and if they don't, it creates them? This doesn't seem quite right.

So let's say I'm booting up a computer from my kernel, which is loaded onto a CD-ROM. Hmm.. As I write this, it's all starting to make sense, so, is this correct?:

- The kernel is loaded into RAM by the bootloader.
- The kernel then can use hard drive reading procedures to talk to the CD-ROM or the hard disk. It would have to interpret the hard disk as the same filesystem format as the bootloader interpreted it as, when loading the kernel into RAM.
- The kernel can then access the various folders and files which are in the disk which the kernel was loaded from.

Please tell me if this is correct :)

In this case, though, how does the bootloader know what is code, and to load into ram, and what is data (such as the /bin, and /etc folders), so to not load those into ram? In other words, when the bootloader (say, Grub), is loading the kernel from the hard drive into ram (by searching for the 0xbadb002 magic number), how does it know when it's got to the end of the kernel program which it wants to load to ram, and into the beginning of data?

Sorry that this question was kind of all over the place, I hope it makes some amount of sense.


Top
 Profile  
 
 Post subject: Re: What generates operating system files/folders, and when?
PostPosted: Tue Oct 29, 2019 9:41 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3590
Location: Chichester, UK
The installation program creates the directory structure when you first install the operating system. This is apart from some "files" that are created when the system boots (e.g. those in /proc). These don't necessarily exist as physical files on the hard disk but are memory-based or created on the fly.

Of course, as you use the operating system other files will be created/deleted.


Top
 Profile  
 
 Post subject: Re: What generates operating system files/folders, and when?
PostPosted: Tue Oct 29, 2019 10:12 am 
Online
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1209
Execution logs from emulation (for example, to know which standard or 32-bit addresses -probably special/hardware-, which I/O ports, in which order, your BIOS firmware or native drivers call) so that for example you can derive a native driver from calling INT 10H video BIOS for each mode it is capable of.

Also, a file that will write a single char every time you press anything on your keyboard.

I have a 512-byte log file that has no fragmentation, that I just find with the partition information with PIO ATA READ SECTOR(S) (in the second IDE partition). I just type in the screen and if I run the command "dsklon", a function that accesses disk for that file gets enabled.

I could add code to that function to log the ports accessed by the emulated BIOS for being able to execute that sequence in protected mode without the BIOS, only by the log of memory and I/O ports as if it was an array of values to write and their addresses.

_________________
http://190.53.102.175/api/doc/ (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Top
 Profile  
 
 Post subject: Re: What generates operating system files/folders, and when?
PostPosted: Tue Oct 29, 2019 12:28 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 290
Leaving what ~ said off to the side for the moment (does anyone here understand him anymore?), the kernel image is a file in a file system. The bootloader loads that file into memory. It is important to separate between the kernel image file and the kernel image in memory. How the kernel gets into memory is mostly immaterial. PXE boot for instance loads it via TFTP from a server. The kernel can then boot from whatever medium is desired, maybe even with NFS as root.

For boot loaders there are mostly two approaches: Either, like GRUB or syslinux, you parse the file system structure (even though each of syslinux, isolinux, and extlinux only understand a single FS each), and parse that at run time, or, like LILO, you save a list of blocks somewhere. The LILO installer asks the OS where the kernel image is on disk and writes the blocks into the bootloader itself, in order to load the image using BIOS.

Fundamentally, the boot loader is a pretty simple program that can only ever load predetermined files into memory. If that file happens to contain a kernel, then that's a happy coincidence, made possible by the intelligence of the configuration.


Top
 Profile  
 
 Post subject: Re: What generates operating system files/folders, and when?
PostPosted: Tue Oct 29, 2019 5:19 pm 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 406
Location: Ukraine, Bachmut
Quote:
Fundamentally, the boot loader is a pretty simple program that can only ever load predetermined files into memory. If that file happens to contain a kernel, then that's a happy coincidence, made possible by the intelligence of the configuration.

on fossilized systems, stuck in 70ths it is, but it's not the only possible approach, let's not confuse people, making them think one's beloved approach is the only one, whereas it's not even good one. OS loader could be way more sophisticated and dynamic in what it should do. and that is set with different ways of configuring/influencing it. for example user input during boot, can direct it to some special boot flow - say "safe boot" or "last known good" option. or different flags, it should take into account and pass them to the kernel - PAE, or 3+1 vs. 2+2 split on 32 bit systems. another thing of its not so simplicity is what set of functionality it should be capable of - it might support a bunch of file systems, though read only, - to be more robust in accessing storage. also - how many and what files it should load, as well may be fully configurable and not hardcoded. say, it loads "registry", some kind of it, and looks there for list of needed modules to load - kernel, boot drivers. this set is configured outside of the loader - at the OS runtime by the OS and driver installation procedures. and so on. also switching virtual memory on, some CPU configuring (TLBs, caches, branch predictor) and building initial mappings - are its responsibility, not so "simple".

Quote:
and then realised that I actually don't know what bit of software creates the OS filesystem

as iansjack said, it's installation process creates this during installation. I'll add, that it copies it from the installation source medium (your "live" CD e.g.), it serves as a template, and is generated during build or packaging process, when you create your OS installation package. you put there relevant pieces (here, having modular OS is way more superior, than monolithic), they contain different sets of files, not necessarily all needed for a particular installation, but when the installation process is running, it chooses what's needed, what user confirmed he/she wants to install. this way, you create an OS installation instance on a target medium.

Quote:
Does the kernel, when booting up, check that all of these folders/files exist, and if they don't, it creates them? This doesn't seem quite right.

no, it doesn't, it doesn't need to. it searches for some predefined places, where persistent configration is stored - registry in Windows for example. there, paths to other places might be specified (dynamically and configurable), so they don't need to be hardcoded. basically, your OS holds its persistent state information in some form of internal database, it's enough. depending on the development level of the OS, it could be some well thought out database, or just a messy set of text files, with different syntax in each, the only common thing between which is extreme unfriendliness to both the OS and human. xD

Quote:
- The kernel then can use hard drive reading procedures to talk to the CD-ROM or the hard disk. It would have to interpret the hard disk as the same filesystem format as the bootloader interpreted it as, when loading the kernel into RAM.

yes, but with full support of the FS and other entities. meaning OS loader only needs to have read only support, it works in a restricted environment (interrupts masked, MMU can be off, no virtual memory etc), it doesn't need to support all features of the FS, and they might be numerous, just a read support to be able to read files out there. the same goes for registry (internal OS's configurational database), the OS loader needs to understand it as little as for being able to get out there the needed info on what to do, whereas OS needs to support everything, the full support - write, journalling (logging) etc.

Quote:
- The kernel can then access the various folders and files which are in the disk which the kernel was loaded from.

obviously.

Quote:
In this case, though, how does the bootloader know what is code, and to load into ram, and what is data (such as the /bin, and /etc folders), so to not load those into ram? In other words, when the bootloader (say, Grub), is loading the kernel from the hard drive into ram (by searching for the 0xbadb002 magic number), how does it know when it's got to the end of the kernel program which it wants to load to ram, and into the beginning of data?

Sorry that this question was kind of all over the place, I hope it makes some amount of sense.

generally speaking, it knows what to load and what not to, by following some protocol/convention. grub does this too, but it's too generic, so you need to adapt to it, whereas an OS specific loader is more specific, more fine grained in its knowledge about the kernel needs - it follows conventions and satisfies demands of the particular OS. say, our loader knows, that it needs to read SYSTEM registry hive file (a piece of the aforementioned OS config database), so it has 1) understanding of a set of supported FSs. it asks firmware to read drive on the block (sector) level. then it identifies the FS, if it belongs to the supported set, it proceeds parsing its internal metadata, asking FW to read more sectors. finally it searches for SYSTEM hive file and if successfully found, loads it. then it parses its structure and finds the list of drivers of the boot group - the drivers it needs to load for the OS to start, to avoid cyclic dependency. with this approach, even the kernel itself may be specified. so, it knows a full path for every module it needs to load. it loads them. and then also makes some in RAM modifications to them - base relocations, zero filling of uninitiliazed variables - if this is its responsibility, it could be. this is how it knows.

_________________
future big goal: ANT - NT-like OS for mips, arm and x86.
current smaller goal: efify - UEFI for a couple of boards (mips and arm).


Top
 Profile  
 
 Post subject: Re: What generates operating system files/folders, and when?
PostPosted: Wed Oct 30, 2019 7:16 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 414
j4cobgarby wrote:
Does the kernel, when booting up, check that all of these folders/files exist, and if they don't, it creates them? This doesn't seem quite right.
Actually that's quite common approach. The Linux kernel for example creates a filesystem in RAM on boot, and it fills up with entries found in the initrd, which is usually a cpio archive, and is loaded by the boot loader. For loading the kernel and the initrd the boot loader does not have to interpret file systems. For example in Xv6, the kernel simply starts at sector 2, outside of the partitions. The boot loader is unaware what's inside an initrd, so it does not need to know it's filesystem either. For the loader, both the kernel and the initrd are just unorganized bit-chunks, with the only exception that the loader has to find the entry point of the kernel, but that's all (this is usually done by reading one pointer from the beginning of the file, either ELF, PE, or a.out. No need to investigate the kernel's executable deeper, however the Xv6 loader does and it copies out the segments from the kernel into their specified addresses).

j4cobgarby wrote:
- The kernel can then access the various folders and files which are in the disk which the kernel was loaded from.
They are not on the disk, rather in memory in a so called VFS structure. It only comes later when drivers can access sectors on disks and the appropriate FS driver interprets those sectors as a filesystem an attaches them to the VFS. Here there's a difference in monolithic and micro-kernel architecture, because the former can boot without an initialized VFS (and may mount the root filesystem directly from disk), however for a micro-kernel a boot-time VFS is a must as the disk driver file has to be loaded, and it can't be from the disk (obviously). So there's an extra abstraction layer here, which in early booting phase can only be eliminated for monolithic kernels (but they usually don't do that, see Linux).

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: What generates operating system files/folders, and when?
PostPosted: Fri Nov 08, 2019 6:54 pm 
Offline

Joined: Mon Jun 13, 2016 2:25 am
Posts: 19
Location: Sydney
j4cobgarby wrote:
This is probably a really simple question, but I'm just beginning to think about hard drive access in my operating system and then realised that I actually don't know what bit of software creates the OS filesystem (for example /bin, /etc, etc.. on linux), and when it's generated.

Speaking very generally, the necessary file-system structure for running an OS would typically be created during it's installation process.

j4cobgarby wrote:
Does the kernel, when booting up, check that all of these folders/files exist, and if they don't, it creates them? This doesn't seem quite right.

The operating system has to assume that certain files needed for operation ( fonts, gui static content, etc ) are present in the file-system, if they're not present then loading the OS can't continue in a reasonable fashion. If they won't be created at this stage, booting will just fail. This is speaking very generally though, it depends on what kind of OS we're describing.

j4cobgarby wrote:
- The kernel is loaded into RAM by the bootloader.
- The kernel then can use hard drive reading procedures to talk to the CD-ROM or the hard disk. It would have to interpret the hard disk as the same filesystem format as the bootloader interpreted it as, when loading the kernel into RAM.
- The kernel can then access the various folders and files which are in the disk which the kernel was loaded from.

Close, but not 100% correct. Again, very generally speaking, the kernel is loaded into RAM by the bootloader. The kernel can then load its own drivers to talk to the CD-ROM, HDD and other hardware. From there the kernel isn't limited in how it interprets the contents of physical media. Broadly speaking, in order for disk media to be bootable it needs to have a specific filesystem partition loaded at a certain physical location to be interpreted by the bootloader. The rest of the physical media can be formatted in any manner you like, as long as the OS can load drivers to read it in a meaningful way.
I like to think of the bootloader and kernel as being different programs entirely. I know that this simple mental model does not accurately describe all real-world cases, but it aids in understanding academic examples like this.

j4cobgarby wrote:
In this case, though, how does the bootloader know what is code, and to load into ram, and what is data (such as the /bin, and /etc folders), so to not load those into ram? In other words, when the bootloader (say, Grub), is loading the kernel from the hard drive into ram (by searching for the 0xbadb002 magic number), how does it know when it's got to the end of the kernel program which it wants to load to ram, and into the beginning of data?

Following the example I've been following, the 'data' you've described above ( the /bin, and /etc folders ) will not be part of the kernel executable at all.
Think of the bootloader as an application that loads at boot time to bootstrap the kernel. It reads the kernel executable from the boot partition and loads it into RAM as you noted in your question. It does this by interpreting the format of the kernel executable and loading it according to the standards of that format. One common executable format is ELF, which specifies a clear way to demarcate executable code sections from data. GRUB contains code for loading an ELF executable into memory, there's a page in the wiki with more information on how an ELF executable is loaded and a wealth of information elsewhere too.
Once the kernel is loaded it can decide for itself, by loading media drivers, how it loads the rest of those files and folders. They'll typically be stored somewhere on your installation ISO.

I hope that helps!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group