OSDev.org

Most cards in use now are SDHC. For both SDSC and SDHC cards, FAT (not exFAT) is the standard defined and de facto used FS. exFAT is only for SDXC cards. the particular type of FAT used for the former - 12, 16 or 32 is defined judging by the cluster count, depending on the volume size and cluster size combination - that's why I said I meant the whole FAT family. SDXC cards are too expensive for OS dev experiments. hoping your users would use them for your OS is as brave as hoping they would install your OS on their brand new laptops with i7. This is what I meant - USB2/3 sticks and most SD cards are formatted as FAT16/32, that's why it's cool to have it as the first FS to implement. >_>

choosing what will be a main FS, depends on people's tastes. so asking about what my taste should be, means letting someone decide on what you could better do on your own. some would want extN, some wouldn't. maybe the author should just read about FSs? collecting some information and having thinking sessions about it.
a thing to note is that the main FS should be (eventually at least) pretty capable, and it means it's something a little bit too complex to start with. the ext2->3->4 route may be a choice here, but again, - only if one wants to have extN as their main FS. I'd be happier to have NTFS (or a simpler lookalike, inspired by the design of the original, where it's known) or JFS or BFS, XFS even. the latter has 80000+ LoC in linux (had, when the articles about it, I read, have been published). at least it's good for plans/dreams.

Just to be clear, UEFI is happy without FAT, see any Apple computer.
Also there's nothing in the USB mass storage or SDHC / SDXC specifications that would explicitly prevent using other filesystems than FAT. (And indeed, you can use ext4 on an USB-stick or on an SD card without problems.)

As a beginner filesystem, I'd also suggest TAR, which was designed originally for tapes, therefore it's extremely simple. An i-node record followed by data records for every file, that's all. Also tools to create and read .tar images are widespread and available on all platforms (Win, Linux, MacOSX, *BSD... name it).

Cheers,
bzt

just to be clear, bzt, again, - this is called incompliance. UEFI requires, that any compliant implementation had support for FAT, but doesn't prohibit supporting anything else, so if apple's UEFI doesn't support FAT, it's not UEFI, if does and also supports somethingelseFS - it's OK, but then you repeat your misunderstanding or double standards, I don't know, choose by yourself.


of course you can format your removable storage device with anything, even with your FS, destroyed by the Brendan's criticism, but if you will force your users to do that, they most probably, will say "nope". and again - the SD specification explicitly requires FAT as a FS, anything else may or may not work properly with SD cards, so again, - you keep repeating your mistakes, despite having been pointed some time ago, that you are wrong. please, don't confuse others.

Let's be honest, anyone that's willing to run an OS that isn't the one that shipped preinstalled on their device is not likely to be scared off by having to format an SD card with something other than FAT32. The standard specifies FAT32 because the standard was designed with the idea that it would be used for transfer of data between devices running various operating systems, including devices with fairly limited computing power, and *not* as boot storage for a full-up computer. If somebody is inclined to boot off of an SD card, they won't have any problem with reformatting it to the needed filesystem.

The fact that the UEFI specification mandates read support of a FAT-like filesystem is an irrelevance when writing an OS. Sure, it may be necessary when you get to the stage of a self-hosting OS (at which stage it should be trivial to add FAT support), but not when one is starting out. There are plenty of tools on your development platform to read and write FAT disks.

This is the worst possible argument for choice of filesystem. Choose an interesting one, a useful one, one with plenty of features. Don't worry about the ESP.

These stand for you, don't you think? Please don't confuse others with your contradictional statements.

Is it just me who feels a little contradiction here? SD cards ARE removable storage devices. And again, when I asked you to show me where exactly the SD specification mandates FAT, you couldn't show it to me. What you showed was an addendum, a recommendation, not part of the basic low sector level specification (which BTW does mention FAT as an example yes, but just as an example when to use certain commands, there's nothing "required" about that; furthermore those commands are reportedly working with other file systems' sectors as well).

Literally millions of people are using USB-sticks and SD cards with ext4 without the slightest problem (for example all raspbian Linux users). Nobody ever reported an issue, end of discussion.

My FS is fine, thank you. Actually if anything, it's thriving, has even better performance than before as my OS supports write bursts now (on Raspberry with SDHC cards ), and also has a FUSE driver which you can use under Linux, MacOSX and Windows as well. On the other hand, last time I checked Brendan was not a member of this forum any more. Just keeping you updated.

Cheers,
bzt

it is just you, being needlessly stubborn, ignoring facts. there is no contradiction - you can format it with another FS, other than FAT, but you cannot guarantee, it will work properly then.


what I showed to you is called "SD Specifications Part 2. File System Specification" and it's not "an addendum, a recommendation, not part of the basic low sector level specification" - it's your delusional fantasies. who told you, that the part 1 - "physical layer" is the only one "real" documentation? low level, mid level, - what are these terms here? where did you get them from? who told you, that commands are everything, that comprises the specification? angel? genie? fairy? no matter who, they lied to you, look at the "General Description" section of the 1st part, page 17 in my item of the Simplified version, there is a "Figure 1-1: SD Specifications Documentation Structure", it shows the specification structure and the accompanying text describes it as well. all those parts are required and constitute the SD specifications and addenda. Part 2 is not an optional "addendum". just like part 3 btw as well, the latter specifies "security". what you've stuck up with is just one part of the set, made freely available in a cut off variant - in no way it's an entire spec. for getting the whole one, with the full part 1 incl, you need to pay money, sign NDA etc.

when the specification talks about something "not mandatory", it clearly states that. let's take for example "SD Specifications. Part A2. SD Host Controller". on the page 14:

see? part 2, on the other hand, "specifies", not "recommends".


literally don't exaggerate. did those "millions" reported "no problem" reports to you? nope. I told you, I faced with such a behavior - the card works well, formatted as FAT by the SD formatter from SDA, but goes nuts formatted as ext4. I had to try all provided images from Pine for the board, thinking I am doing something wrong or images are bad, or there are bugs - several builds, until finally one of the images worked longer, than hanging at the booting and I started to suspect SD card problem. and when I reformatted it back into FAT, it got sane again, and I use it still. how many of those "millions" rpi users threw away cards, thinking they are "counterfeit" or like, when facing problems, whereas it was incompliant card usage result?


if you think installation process for your OS is something of a low priority, then maybe, but I think oppositely, looking at how ARM SBCs people torture themselves with that crappy "flashing" linux by "dd", literally manually putting all pieces, starting from kind-of-firmware uboot, which btw requires putting at specific LBAs, I see, how lame this is without the proper installation. every version of every linux distribution supported then needs to be a separate build and image, because this approach means writing a "disk" as a whole and also has a lot of hardcoded values inside the images - uboot environment variables, addresses in the uboot scripts - if that were modularized and flashed by the installation, it would get so much easier and more elegant. and then, when you don't overlook the installation, you see that having (ironically) write support for FAT for being able to update ESP (which is FAT32 and only it) is a must on UEFI platforms. which are getting de facto the only standard used on modern x86 PCs.

yup, ext2 is very interesting. I repeated several times why FAT is useful. is my understanding of "useful" wrong?

That's interesting! I might implement it myself because it's reputed to be more resistant to corruption than FAT, or at least FAT12/16. Personally, I wouldn't go to ext3 or higher because I do not want high-latency filesystems. I could tell you this horror story about Gvim locking up for 17 minutes simply because I was running a grep/find query on a somewhat large set of files. It was years before the ext3 nutters fixed the default to something sane, which admittedly I could do from the start, but I hear Linux filesystems are still much higher-latency than even Plan 9's weird multilayered userspace cached WORM filesystem. I remember when Linux was responsive and fast: It was before preemptable kernel and fast tickspeed; it was also before ext3. *huff* Enough rant.

About this argument, I haven't read all the walls of text (or even most of them), but...

@bzt: Sorry mate, Apples aren't UEFI, just EFI.

@zaval: Did I ever tell you that memory stick you used to justify standards-complience (in another thread) was probably faked? A flash drive which has been reprogrammed to fake its maximum size can't handle any other filesystem than the FAT(s) it's programmed for. Properly-functioning Flash drives generally don't care what filesystem you put on them. This is all well-known by just-about everyone involved with SBCs and/or unusual operating systems.

I don't see how ext3 adds latency to read operations. Ext3 probably adds latency to write operations (because all logical write operations have to be written to the journal first, then to the disk, then removed from the journal), but reading should even be faster, because of indexed directories (if you have large directories. Most applications try to limit the amount of files created in a single directory).

Also, without specific data, I wouldn't trust assertions like "X is more corruption resistant than Y". In this generality, it is an almost useless statement. What kind of corruption is more or less likely in one system or the other? A head crash will destroy your data as reliably now as it did thirty years ago. And shutting the system down during a write operation is as perilous in ext2 as it is in FAT.

Take this action: Creating a directory. In ext2, you have to allocate an I-node (write a 1 to the I-node bitmap), allocate a data block (write a 1 to the block bitmap), initialize the I-node (write a block there) and initialize the data. Then add the directory entry to the parent directory and update its link count (it just went up by 1). Mess any of this up and you have an orphaned I-node, or a directory entry that doesn't point anywhere, or an inconsistent link count on a directory, or the new directory file is suddenly some other kind of file.

On FAT you have to allocate a new cluster, write an empty directory to it, and add a directory entry to the parent directory. Mess any of this up, and you have an orphaned cluster, or a directory entry that doesn't go anywhere, or an inconsistent directory... remarkably similar.

Maybe you are right about the corruption resistance, but personally, I don't see it. Messing up a cluster chain is probably easier than messing up an I-node, though.

@nullplan: Interesting comments! I think the only hard data I know of is that ext* have duplicate superblocks, so if the first is corrupt, the filesystem can be recovered from one of the others. Also, they're scattered throughout the partition. FAT of course may have multiple FATs as such, but usually has only 2 of them and they're adjacent; meaning they're clumped together in the same physical area of a primitive disk. That can't have helped recovery from localised damage in the 80s & 90s, thus getting FAT a bad reputation which may not be applicable now disk drives scatter sectors across different platters.

I forgot something when writing my last post because it didn't happen in the time period I was thinking about. It may make Linux fanboys' heads explode. In proportion to the time each filesystem was used, I have lost as many files to corruption in ext4 as I did in the very worst released version of ReiserFS. That being the case, perhaps the reputation for reliability of the entire ext series is mostly hubris, or at best dependent on the scattering of allocation data across the disk which is no longer relevant.

The thing I always wonder is: What do you do if your multiple copies of the FAT or the superblock disagree. How do you find out which version is correct? And do I want the code that does the deciding here in my kernel?

Well, thanks for the concern but my head is still very much intact. I do, however, wonder how you manage to break your file systems. I only ever ruined one ReiserFS by kernel panic. I tried many different FSes with Linux over the years, but it turns out that my workload is not particularly hard on the FS, and therefore the differences between the file systems don't matter all that much.

All right, EFI then. My point was (or supposed to be) that the ESP file system is not something that's hardcoded and irreplacable in the firmware; you can change it easily, and you can have a firmware that's 100% EFI/UEFI compliant and 100% EFI API compatible without FAT.

An EFI application that loads files from the ESP is the same, no matter whether the ESP is actually using FAT or ext2 to store those files (as long as the ESP file system has a driver that implements EFI_SIMPLE_FILE_SYSTEM_PROTOCOL, the EFI app wouldn't know, and doesn't have to know which fs is it). I have tested this successfully with a small EFI application that read a text file and printed its contents on the console. I was able to run it on a "normal" desktop and on a Mac without recompilation or anything.

I think you say some nice words about the occupation of the mother of the fs' designer, and you reformat the disk with ext4
Imho that goes into a separate user space application, fsck, which should be run when the fs is not mounted. Even Windows has (or had) such a file system recovery mode.

Cheers,
bzt

dood, I was talking about SD cards and the specifications, that they should comply with. USB flash sticks are mostly formatted as FAT, I meant this - you have support for FAT early, your early users have USB sticks, formatted as FAT, thus they don't need to wipe them out and reformat to try with your OS, everyone is happy. it was a little, but practical objective in favor of having FAT instead of that painfully useless ext2 thing. the farther "discussion" and arguments are pure crap caused by bzt's idiocy. my fault was I got myself in that again.


unbelieveable. tell me, are you trolling or maybe it's something worse? no, you can not have it compatible, since ESP is always FAT32. read the f&cking specification already! here is the excerpts for you, for version 2.7. page 662, section 13.3 File System Format, and no, those aren't "addenda", "recommendations" nor "marketing BS embedded by M$, scientologists and reptils", just a legit part of the spec, read it, damn it:

section 13.3.1 System Partition tells you what ESP is.
13.3.1.1, File System Format tells it for you, especially gifted one, with an emphasis:

come on, I believe you can get it!


so this is your "proof", what ensured you FAT isn't part of the spec? vs. what is clearly written out in the spec. hilarious. okey, if you are so dense, here yet once, this:

is your delusional BULLSHIT! ESP is always FAT32. and FAT is the only required FS to implement. anything else could be "as long as" that protocol is implemented for the FS, but 1) then it's not ESP and 2) not required. if apple marks HFS partitions as ESP, their firmware is not UEFI, if they don't, they just implemented value added stuff. but still their FW does support FAT and moreover, as the spec defines, page 668-669, 13.4 Simple File System Protocol, Description:

See, it is the only one "automatically" handled - those for which the FW should create handles, through which you access those volumes. no BztFS, no ext2, no HFS+, no "any FS".
it's value added whistles, if you, as a FW provider, let the Boot Manager process boot options, created with non-FAT partitions and it's highly unportable. noone else would support your whistles, so for example in the case of apple, it can make troubles for their users wishing to install another OS - it won't work when the OS would want to modify or create its folders in the ESP in the live session during installation for example. say, you try to install you BztOS on iMac and you booted with the FW and now in live session, your user chooses to install BztOS, and your installer wants to create \efi\bzt folder. for putting there bztosloader.efi and bztwolflogo.bmp. and then whoops, the "ESP" is not in fact ESP, apple thought different and pooed in your shows. cheers!

I am stunned by this. for one things people scream about "violating standards" and how bad it is, but for other things the very same people see it as OK and moreover, show standard violation as a "proof" of their truth. "oh, I took that iCrap and it worked just fine without FAT, it's the proof, FAT is not needed" seriously? then who the hell decides what is UEFI, if the spec saying clearly, loudly, spitting in your face "no, f&cking sh1t, you should use FAT32 as ESP" is not?

Ext4 also has multiple copies of the superblock. What if they disagree? Besides, neither the FS' designer nor their mom can do anything about my disk wearing out, as is the course of all things in the universe.

I think the salient point here is "multiple copies" as opposed to "two copies". With multiple copies it is highly likely that all copies but one agree. You take the majority to be correct. With two copies you toss a coin.