OSDev.org

As far as the FATness of the ESP, it appears to be the case that

1) The concept of the ESP was introduced with EFI, but the filesystem to be used was not specified until UEFI came along.
2) For UEFI, the exact filesystem to be used is specified, and the specified filesystem is basically, but not identically, FAT32 (the specification is maintained independently of the original FAT32 specification).
3) Old Intel Macs used EFI with an HFS ESP. Newer Intel Macs use UEFI, and seem to have a FAT32 ESP, but have their own boot method if no UEFI bootloader is found, and in the normal state of the system the ESP is completely empty, so boot falls through to the alternate method, which loads MacOS directly from its own partition. The system will operate completely normally if the ESP is removed completely, with the exception that firmware updates are staged through the ESP and will fail if it does not exist.
4) Nothing I've found so far indicates that the behavior described in 2) violates the spec, though it probably wasn't what the people that wrote the spec intended, and by definition is not part of the spec.

My own observations:

4) Other situtations with an empty or non-existent ESP that would not be out-of-spec, and would be closer to the spirit of the specification than the Mac case can be imagined, such as when network booting on a system with no local disks, so an operating system running on a fully-compliant UEFI machine cannot necessarily count on having an ESP available.
5) An OS in development that is not self-hosting does not necessarily need to interact with the ESP at all (basically letting the host OS do the ESP-wranglng), so FAT drivers do not necessarily need to be written immediately.

nullplan wrote:Ext4 also has multiple copies of the superblock. What if they disagree?

There are more copies, so you can always make a majority vote: the version that's the same in most of the superblocks is the correct one. Of course, all superblocks may differ; but the chance to that is slim, much-much lower than the chance of a single bad sector with FAT. The best would be to calculate a checksum of the allocation info and wrote that checksum in the superblock, that way you could know for sure which superblock is up-to-date (not sure right now, I'd have to look it up, but maybe ext4 have something like that with journaling enabled).

nullplan wrote:Besides, neither the FS' designer nor their mom can do anything about my disk wearing out, as is the course of all things in the universe.

As the smilely suggests, it was just a joke. Never mind, no worries if you didn't get that. Irrelevant.

Cheers,
bzt

@iansjack: That was part of my point, yes. One of my early (and relatively small) ext2 installations had eight copies of the superblock if I remember right.

@zaval: I'm sorry. I'm too willing to fight too.

@bzt: Am I right in understanding I could use UEFI to write my boot image when I natively reassemble it? Did you use different OSs to write your test EFI application to disk to be booted?

I'm a little bit annoyed with UEFI itself, actually. I'm developing a Forth system which doesn't initially need a filesystem for its own purposes, and when it does, I think it could probably use tarfs for quite some time. (Funny thing to write in this thread? ) Having to write and subsequently ignore a filesystem image of minimum size over 32MB for an operating system which will likely be under 32KB is a little annoying. Cross-assembling for bootstrap or recovery, I guess I'd have to use mtools to make a disk image or mount the partition in an OS which supports FAT32. This is not terribly hard, but it's an extra step in the process and it's a step away from something I'd almost achieved: controlling every byte of my build environment. I'm not paranoid, it would just have been nice to have achieved that. Then again, if that's what I want, I shouldn't be developing for complex hardware.

@nullplan: My filesystem usage was also light. I don't recall what, if anything, I did to break the filesystems.

eekee wrote:@bzt: Am I right in understanding I could use UEFI to write my boot image when I natively reassemble it?

Yeah, sure, why not? You can write raw blocks to disks so you can write whatever you want. I haven't tried that as my boot loader only reads the disk, so I only use BLOCK_IO_PROTOCOL.ReadBlocks, but it also has a WriteBlocks method.
This is how my boot loader works:
1. it tries to load the boot image from a file on FS0: using EFI_SIMPLE_FILE_SYSTEM_PROTOCOL (I let UEFI care about the fs)
2. if that fails, then the loader loads the GPT using BLOCK_IO_PROTOCOL.ReadBlocks
3. if it can locate the boot image partition (by checking partition UUID), then it loads the entire partition into memory (should be about 8-16Mb no more)
4. the rest of the boot loader doesn't know nor care where the boot image was loaded from (file or partition), and it interprets the file system image in memory (which could be tarfs for example)

Now an UEFI app could do this in the opposite direction for writing:
1. construct an image in memory with a file system your choosing
2. use BLOCK_IO_PROTOCOL.ReadBlocks to read the GPT, and locate your partition's starting offset and length
3. use BLOCK_IO_PROTOCOL.WriteBlocks to write the file system image in memory to that partition
4. alternatively you could save it to a file on FS0: using EFI_SIMPLE_FILE_SYSTEM_PROTOCOL

eekee wrote:Did you use different OSs to write your test EFI application to disk to be booted?

Well, yes and no. I wrote a small ANSI C tool to generate a multi-purpose disk image which happens to be ESP-compatible as well. That tool is multi-platform, uses libc only, so you can compile it on different OSes. I actually use the "dd" command to write the generated image to a removable media, supported by BSDs, Linux and MacOSX, so it is multi-platform too. Long-long time ago I used rawrite.exe on Windows, but I don't know if it's relevant for the latest versions, or if the built-in diskpart command can write images these days.

eekee wrote:I'm a little bit annoyed with UEFI itself, actually. I'm developing a Forth system which doesn't initially need a filesystem for its own purposes, and when it does, I think it could probably use tarfs for quite some time. (Funny thing to write in this thread? ) Having to write and subsequently ignore a filesystem image of minimum size over 32MB for an operating system which will likely be under 32KB is a little annoying.

Yeah, I agree. But since ESP is mandatory, I'm afraid there's not much you can do about it. But on the bright side, if you have to use UEFI anyway, then forget about tarfs, you can take advantage of EFI_SIMPLE_FILE_SYSTEM_PROTOCOL to provide file system for your Forth interpreter. (Or just read the tar file and handle it as a ramdisk)

eekee wrote:Cross-assembling for bootstrap or recovery, I guess I'd have to use mtools to make a disk image or mount the partition in an OS which supports FAT32. This is not terribly hard, but it's an extra step in the process and it's a step away from something I'd almost achieved: controlling every byte of my build environment. I'm not paranoid, it would just have been nice to have achieved that. Then again, if that's what I want, I shouldn't be developing for complex hardware.

Actually there's a shortcut you could take.

Prerequisites: create an empty ESP (format it with whatever tool you'd like), then copy over 64K worth of 'A's by the name \EFI\BOOT\BOOTX64.EFI. Set the System flag on that file (this is important to tell other fs tools not to move the file). Then with a hex-editor, open your disk image, and find those bunch of 'A's. Remember the offset where the 'A's start.

Normal build process: compile you application, and use "dd" with "conv=notrunc seek=X" to write your app into the place of the bunch of 'A's. This way you won't need mtools nor any FAT32 in your build environment, just a pre-recorded offset. This will work perfectly fine as long as your compiled application does not exceed in size the number of 'A's.

Before I've created my fully-featured disk image creator, I've used a version of this method. I used mkfs.fat, created a partition, copied a file there full of 'A's, then I saved everything from the master boot sector to the start of 'A's into fat32.bin. My build process knew nothing about fat, it simply wrote that fat32.bin before the compiled kernel when it created the bootable disk image.

Cheers,
bzt

Thanks bzt, I especially like the shortcut. I was thinking some machines probably won't implement file writes correctly, but block writes should work fine. (If they don't use it, and regardless of standards, some won't, then they probably won't implement it correctly. Such is the way of the world.)

I went and started coding for the BIOS last Wednesday, going with what I already know. I'll port to UEFI when I get more familiar with the Forth interpreter I'm porting, but I might need to keep the BIOS version. 2 of my 5 development PCs may be too old for UEFI, including my primary development machine, and one is of unknown age. 3 of the 5 are Lenovo and 1 is Gigabyte; the wiki page on UEFI problems explicitly mentions both brands.