Implementing SFS

[0xBADC0DE]

Hi everyone!

I've been reading up about filesystems and trying to decide which one to implement in my operating system, and I've decided to go with SFS. I've never written a filesystem before, so any help will be greatly appreciated!

The following is what I understand from the SFS wiki page (key areas):
- the super-block contains information about what filesystem is contained within the current partition and contains information relating
to the size of each area
- the data area grows as files (data) are added to the filesystem
- the index area contains information about the stored files (filename, extension, date created etc)

Is this correct? Please correct me if I'm wrong.

As for the block size, do I choose my own size? Would 4KB be a good block size?
For the magic number, can this be placed anywhere inside the bootloader (I'm using GRUB and multiboot). Or do I need to place it at a specific offset? If so, how do I specify the offset?
And for the checksum, how would I go about calculating that?

PS: I do not yet have a disk driver, so I am not looking to store the files on disk, but rather in memory.

If there's anything else I need to know about writing a filesystem, please let me know! Thanks!

[Brendan]

Hi,

The following is what I understand from the SFS wiki page (key areas):
- the super-block contains information about what filesystem is contained within the current partition and contains information relating
to the size of each area
- the data area grows as files (data) are added to the filesystem
- the index area contains information about the stored files (filename, extension, date created etc)

Is this correct? Please correct me if I'm wrong.

Yes - mostly the data area is near the start of the volume and the index area at the end of the volume, and they both grow towards each other (causing the free area between data area and the index area to shrink).

As for the block size, do I choose my own size? Would 4KB be a good block size?

The spec says "For media that is organized into "sectors" (floppy disks, hard disks, etc) the block size must match the size of a sector.". If you're storing the file system in RAM then this doesn't apply and you'd be able to choose anything you want; and (on 80x86 machines) 4 KiB would probably be an excellent choice (because it matches page size).

For the magic number, can this be placed anywhere inside the bootloader (I'm using GRUB and multiboot). Or do I need to place it at a specific offset? If so, how do I specify the offset?

For SFS, the magic number must be at offset 0x1AC in the superblock (in the first block of the volume). GRUB doesn't really belong in any file system (GRUB belongs in the space before the start of the first partition/volume, and file systems belong in partitions/volumes).

Note that I doubt GRUB supports loading files from (booting from) SFS; and SFS isn't really intended for that anyway (it's mostly intended for "sneakernet file transfer" - e.g. where you wipe a floppy disk clean and format it as SFS, copy a bunch of files to it, then shift the floppy to a different computer to read the files off of it).

And for the checksum, how would I go about calculating that?

That's a common "8-bit sum of bytes", so if you add the values at offsets 0x01AC to 0x01BD (inclusive) and AND it with 0xFF you get zero. To calculate the checksum you'd do the reverse - e.g. start with zero and subtract the values at offset 0x01AC to 0x01BC (inclusive) and AND it with 0xFF then store the result at offset 0x01BD.


Cheers,

Brendan

[0xBADC0DE]

Thanks for the reply, Brendan. I understand better now.

EDIT: I've been reading about initrd and that GRUB can load the "initrd.img" file using a command like:

Code: Select all

module  /boot/myos.initrd

But the problem I have still remains. Once I load the initrd and my operating system has finished setting up everything, how do I load my filesystem from the ISO file as mentioned below?

So, I'm looking to store the filesystem in memory as I previously mentioned. The way I thought of doing this, was to store all the files and their information in one single file, which would be unpacked by my operating system and then stored in memory instead of on disk, kind of like how Linux live works (I think). Looking through the Ubuntu ISO file, I saw a file named "filesystem.squashfs". That's where I got the idea from.

But the problem is, how do I load my custom "filesystem" file (like "filesystem.squashfs") into memory so that it can be unpacked by my operating system and the files can be used? The custom "filesystem" file would be stored in an ISO file with my operating system kernel inside. How would I go about loading a file that is store inside an ISO file? Would I have to read the ISO file? Or specify the disk image to qemu with -hda (which I don't want to do)?

Thanks

[Brendan]

Hi,

But the problem is, how do I load my custom "filesystem" file (like "filesystem.squashfs") into memory so that it can be unpacked by my operating system and the files can be used?

The same way you get a kernel in memory - get the boot loader to load it.

GRUB/multi-boot has full support for this. You just set it up in GRUB's configuration ("menu.cfg"?) and the multiboot information given to the OS will include a flag saying that none or more "boot modules" were loaded, plus a count saying how many were loaded, plus a list of the starting and ending physical addresses of each module (and an arbitrary string that you can use for whatever you like).

Or specify the disk image to qemu with -hda (which I don't want to do)?

I wouldn't necessarily trust Qemu's multiboot support (and to me it defeats the purpose of using an emulator like Qemu anyway - e.g. trying to test an OS but then deliberately failing to test if the OS works with a real boot loader); but there is a command line option ("-initrd") that can be used to tell Qemu to load one or more boot modules (in addition to the kernel).


Cheers,

Brendan

[0xBADC0DE]

Ok, so I'm using GRUB to load a module now. I have

Code: Select all

module /modules/filesystem

in my "menu.lst" file.

And in kernel.c I have

Code: Select all

  multiboot_info_t* mbinfo = (multiboot_info_t*) mb_struct;
  unsigned int module_address = mbinfo->mods_addr;

Now isn't this the address of the list of modules, and not the individual module itself (or does it not matter since I have only loaded one module)?
How do I access the file (my filesystem in this case)? I tried reading the first 3 bytes at the "module_address" variable, but got a bunch of garbage. The first 3 bytes in the filesystem file are "SFS" (just for testing purposes).

I thought about using the

Code: Select all

multiboot_module_t

struct, but I am not sure how to use that. Maybe accessing the

Code: Select all

mod_start

field in that struct would give me the address of the module?

[Brendan]

Hi,

Ok, so I'm using GRUB to load a module now. I have

Code: Select all

module /modules/filesystem

in my "menu.lst" file.

And in kernel.c I have

Code: Select all

  multiboot_info_t* mbinfo = (multiboot_info_t*) mb_struct;
  unsigned int module_address = mbinfo->mods_addr;

Now isn't this the address of the list of modules, and not the individual module itself (or does it not matter since I have only loaded one module)?

It'd be the address of an array of structures; where each entry in the array/each structure contains several fields (32-bit starting physical address, 32-bit ending physical address, etc). You can find this documented in the multiboot specification.

How do I access the file (my filesystem in this case)? I tried reading the first 3 bytes at the "module_address" variable, but got a bunch of garbage. The first 3 bytes in the filesystem file are "SFS" (just for testing purposes).

You'd mostly want to check the flag to determine if the "mbinfo->mods_addr" and "mbinfo->mods_count" fields are valid, then determine the number of modules/number of entries in the list (from "mbinfo->mods_count"), then have a loop ("for each entry in the list {...") that checks each entry to see if the module it points to is the file you want. This checking would probably start with making sure the module is long enough for a valid header for the file (e.g. "end - start >= 3 bytes" if you're using the first 3 bytes of the file as the header) and then check the header (including any checksums, etc) to see if it's the module you want.

I thought about using the

Code: Select all

multiboot_module_t

struct, but I am not sure how to use that. Maybe accessing the

Code: Select all

mod_start

field in that struct would give me the address of the module?

If this "multiboot_module_t" is example code provided by GRUB developers that matches the official multiboot specification; then I'd assume it has "mod_start" and "mod_end" fields containing the 32-bit physical addresses that the specification describes.


Cheers,

Brendan

[0xBADC0DE]

This is my current code:

Code: Select all

        // Get memory address of loaded GRUB module
	unsigned int module_address = mbinfo->mods_addr;

	multiboot_module_t modules[mbinfo->mods_count];

	// Stored the start and end address of the module in the struct
	for (unsigned int i=0; i<mbinfo->mods_count; i++) {
		modules[i].mod_start = module_address + (i*4);
		modules[i].mod_end   = module_address + ((i+1) * 4);
		modules[i].cmdline   = module_address + ((i+2) * 4);
		modules[i].pad		 = module_address + ((i+3) * 4);
	}

	printf("Modules[0] start addr=%d\nModules[0] end addr=  %d\nsizeof(modules)=%d\n", modules[0].mod_start, modules[0].mod_end, sizeof(modules));
	printf("Modules[0] char=%c\n", modules[0].mod_start);

"mbinfo->mods_count" gives "1"
"mbinfo->mods_addr" gives "166464"

As you can see, I am trying to create an array of structs containing each module's start and end address. I'm looping through each module, as you suggested, and trying to store the information in the struct, adding an offset to "module_address".

The second to last printf prints

Code: Select all

     Modules[0] start addr=166464 (ie. same as the mbinfo->mods_addr, which I assume to be correct)
     Modules[0] end addr=166468
     sizeof(modules)=16

And the last printf prints out an "@". I was expecting this to print out "SFS", but my code above must be incorrect. As I mentioned, the first 3 bytes should be "SFS".

Sorry for all the questions
Thanks

[Brendan]

Hi,

This is my current code:

Code: Select all

        // Get memory address of loaded GRUB module
	unsigned int module_address = mbinfo->mods_addr;

	multiboot_module_t modules[mbinfo->mods_count];

	// Stored the start and end address of the module in the struct
	for (unsigned int i=0; i<mbinfo->mods_count; i++) {
		modules[i].mod_start = module_address + (i*4);
		modules[i].mod_end   = module_address + ((i+1) * 4);
		modules[i].cmdline   = module_address + ((i+2) * 4);
		modules[i].pad		 = module_address + ((i+3) * 4);
	}

	printf("Modules[0] start addr=%d\nModules[0] end addr=  %d\nsizeof(modules)=%d\n", modules[0].mod_start, modules[0].mod_end, sizeof(modules));
	printf("Modules[0] char=%c\n", modules[0].mod_start);

"mbinfo->mods_count" gives "1"
"mbinfo->mods_addr" gives "166464"

As you can see, I am trying to create an array of structs containing each module's start and end address. I'm looping through each module, as you suggested, and trying to store the information in the struct, adding an offset to "module_address".

There's multiple issues here.

First, there's a problem with the maths - e.g. "module_address + ((i+3) * 4)" looks like it was supposed to be "module_address + ((i*4) + 3)".

Second, there's a problem with pointer dereferencing - e.g. if "module_address + (i*4);" correctly calculates the address of the data you want, then "modules.mod_start = module_address + (i*4);" will store the address in "modules.mod_start" and won't store the data you want in "modules.mod_start".

Third, you're not using the language's type system effectively. For example, if you used "void * module_address" (instead of "unsigned int module_address") the compiler will know that the variable "module_address" contains the address of something and not just an integer value, and if you used "multiboot_module_t * module_address" the compiler would know that the variable "module_address" contains the address of one or more "multiboot_module_t" structures.

Fourth, if GRUB says there's no modules your code will copy nothing and then assume that it copied at least one, and will then blow up.

Fifth, GRUB already gave you a list of "multiboot_module_t" structures, so why are you copying them to create your own list of "multiboot_module_t" structures? Assuming virtual memory is the same as physical memory (e.g. everything identity mapped or paging not enabled, and segmentation not used); your code could delete all the buggy copying and just use "multiboot_module_t * modules = (multiboot_module_t *) mbinfo->mods_addr;".

Sixth, your code to print stuff has the similar bugs - using the address of some data where you wanted the data and not the address. For example, "printf("Modules[0] char=%c\n", modules[0].mod_start);" will convert the 32-bit physical address of the module into a character (mostly likely discarding most of it so it fits in 8 bits) and won't read a byte/character from the memory at that address.

Based on all of these bugs (combined with being unable to recognise some of them yourself), I'd be tempted to suggest that you might benefit from gaining more experience with C (especially pointers) before you start writing an OS in C.


Cheers,

Brendan