OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Oct 15, 2019 11:02 am

All times are UTC - 6 hours




Post new topic This topic is locked, you cannot edit posts or make further replies.  [ 30 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Wed Jun 19, 2019 1:25 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1162
quadrant wrote:
I plan on using an SD card as my "disk" (i.e. persistent memory). What should I keep in mind when designing a simple file system with regards to sending read and write requests to the SD card?
You can even start with FAT32 LBA.

You can format the disk or a partition, create big files with an ID string at the start, and record the position of your files. Since they are defragmented, you just need to know up to which point you have written them.

You can also create empty files and write a string with the sector number of the file at the start of the sector block, so that you can see if in fact you created an unfragmented file in your new test partition.

create_big_file.c
Code:
#include <stdio.h>

//NOTE: These two libraries seem to need to be together
//      to avoid an error message that says
//      "warning: incompatible implicit declaration of built-in
//                declaration of built-in function 'memset'":
//
//      This warning occurs with gcc.
///
#include <stdlib.h>
#include <string.h>

//This is a macro to easily access the first command line
//as the file name the program should use as a big file
//to create:
///
#define creatFileName   argv[1]
#define sectors512Count argv[2]

//(bytes_per_sect*sects)-zero_addressing
///
  #define creatFileSize____512MB 512*1048576-1
  #define creatFileSize____20GB0 39070080
  #define creatFileSize____875GBbytes   939524096000
  #define creatFileSize____875GB4ksects 229376000
                                               //for completing aligned/full file bytes



int main(int argc, char *argv[])
{
//Here we just declare the variables
//to handle this simple big file:
///
  FILE       *fHandle;
  const char *fName   = "hard_disk.img";
//  long numsects=39070080;
  long int numsects=creatFileSize____875GB4ksects;
  void *buff;
  unsigned long sectnum=0;


  if(argc<3)
  {
   printf("ERROR: Please specify a file name and a number of 512-byte sectors\r\n");
  }

//Let's open (and create or overwrite)
//the specified file name:
///
  if(!(fHandle=fopen(creatFileName, "wb")))
  {
   printf("ERROR: Couldn't open %s\r\n", creatFileName);
   return -1;
  }

  if(!(abs(numsects=atol(sectors512Count))))
  {
   printf("ERROR: Sector count (%ld) must be greater than 0\r\n", numsects);
  }


// fseek(fHandle, creatFileSize____20GB0, SEEK_SET);
// fwrite("", 1, 1, fHandle);
// fwrite("", 1, 39070080, fHandle);

buff=malloc(512);

do
{
  memset(
         buff,  //Buffer to zero out
         0,     //(int)(unsigned char) value to write (0 in this case)
         512   //Number of bytes to write
        );
  ltoa(sectnum, buff, 10);
  fwrite(buff, 1, 512, fHandle);
  sectnum++;
}
while(--numsects);



free(buff);
fclose(fHandle);

return 0;
}


_________________
http://190.53.102.175/api (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Wed Jun 19, 2019 3:05 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 1656
bzt wrote:
Octocontrabass wrote:
Additionally, Part 2 does not "recommend" the filesystem, it requires the filesystem.
Despite of this "requirement", I can read and write an SD card with raw data. I had absolutely no problems using my own, totally custom file system either. Works like a charm, and I'm still using the same 4 or 5 years old cards without any signs of shortened life time.

The requirement is for the host, not the card. The card can support whatever it wants, as long as it works with the mandatory filesystem. I'd expect most cards don't care what filesystem you're using, but the few that do are where you'll have problems. Only by following the specs can you guarantee those cards will work with your software.

bzt wrote:
The card does not examine file system structures.

Your cards do not examine filesystem structures, but this does not mean no cards examine filesystem structures. (Did you look at Part E7, which describes cards that must examine filesystem structures?)

bzt wrote:
I'll keep using partitions and better file systems on the SD card without probs in the meantime ;-)

...Until you get a new SD card and start having problems. :roll:


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Wed Jun 19, 2019 2:41 pm 
Offline
Member
Member
User avatar

Joined: Tue Mar 06, 2007 11:17 am
Posts: 1162
You can use a non-fragmented file as a partition inside a regular disk.

You can slow-format a partition, create a huge file as your test partition, record the start sector of that file (and its size in raw sectors). Then you can also verify the data you write with a hex editor like HIEW32 to see if you are writing it properly.
------------------------------------------------

The fact that Windows compresses NTFS files that are just zero-filled down to 1 sector/cluster, shows that they think about them because they have important, valid uses.

But in this case what we need is a big file that is non-fragmented and uncompressed, be under FAT, NTFS, ext filesystems, or whatever.

_________________
http://190.53.102.175/api (My OS compatible with DOS)

(udocproject@yahoo.com)
-----------------------------
IP for hosts file (all domains):
190.53.102.175 api.exe


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Thu Jun 20, 2019 8:31 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 389
Octocontrabass wrote:
but this does not mean no cards examine filesystem structures.
Just for curiousity, how does a card do that, exactly? How can a card detect if a sector written contains FAT cluster addresses and not an Inode list for example?

And what would a card do if the sector is explicitly flagged to contain DIR entries by CMD20? Would it refuse to handle that sector as such if it contains not a FAT directory but an ext4 directory for example? I seriously doubt that any card would do any examination if a sector is explicitly flagged.

This is exactly the same as UEFI stating that ESP must be FAT formatted. Apple has shown that it ain't true, just a paid marketing bullshit, as all Macs use HFS+ instead of FAT.

Octocontrabass wrote:
...Until you get a new SD card and start having problems.
In which case I just throw away that card, buy a different brand, and the rules of free market takes care of the rest ;-)

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Thu Jun 20, 2019 9:45 am 
Offline
Member
Member

Joined: Wed Oct 30, 2013 1:57 pm
Posts: 304
Location: Germany
bzt wrote:
This is exactly the same as UEFI stating that ESP must be FAT formatted. Apple has shown that it ain't true, just a paid marketing bullshit, as all Macs use HFS+ instead of FAT.

It's just that Apple doesn't claim compliance with UEFI, because Macs are EFI. Still, EFI defines FAT32 as the only filesystem you must support, which Macs are capable of booting from (IIRC if no HFS+ partition is found, for example). So they are complying, because they implement what the spec requires them to.

Just because something seems to work doesn't mean that it is any good in terms of correctness. So point remains that the spec says you should stick to FAT (I don't know anything about this, but I'll assume for the sake of argument that this is true). Quite frankly, I do not think that anyone needs an explanation for why correctness is important, so let's leave it at that.

_________________
vineyard, a x86 OS.


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Thu Jun 20, 2019 3:27 pm 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 396
Location: Ukraine, Bachmut
This gets hopeless.
bzt wrote:
Octocontrabass wrote:
but this does not mean no cards examine filesystem structures.

Just for curiousity, how does a card do that, exactly? How can a card detect if a sector written contains FAT cluster addresses and not an Inode list for example?

And what would a card do if the sector is explicitly flagged to contain DIR entries by CMD20? Would it refuse to handle that sector as such if it contains not a FAT directory but an ext4 directory for example? I seriously doubt that any card would do any examination if a sector is explicitly flagged.

What do you think can prevent the card's controller from reading FAT? Inside any SD card, a Cortex-M something MCU is sitting and running its FW, that understands FAT. It can use FAT as it wants for its operation, given every SD card vendor unlike you, got it - FAT is a part of the specification, so they write their FWs to take advantage of this.

As of what card could do with ext4 instead of FAT? Here, is the example of what an SD card can return in such cases. This supposed to be some perl file from the debian packaging system, that should work. but it apparently didn't. I faced this last summer, on a freshly bought ARM SBC. Look at it - is this perl? :lol:
Quote:
nless(XXX){}

}

}

}

}

1;

^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;^@;

After reformatting this card to FAT, it became a well behaved card and even linux can interoperate with it. SD Association Formatter didn't find any problems with this card. So how is it possible in your world, where specification requirements, that you don't like become "just marketing bullsh1t"?

Quote:
This is exactly the same as UEFI stating that ESP must be FAT formatted. Apple has shown that it ain't true, just a paid marketing bullshit, as all Macs use HFS+ instead of FAT.

Seriously? so breaking standards becomes a good doing when it's not "M$"? This is so lame and smells double standards, that it's not worth of paying attention to. But one thing yet once - UEFI doesn't prohibit using other FSs, it states that FAT must be supported and if FW suppliers have intention to use other FS, they should materialize the FS through File System Protocol. So if apple makes ESP formatted not as FAT, it's not ESP, and apple just makes incompliant cr4p. If it just adds support to their HFS+ in addition to the FAT support, through the defined mechanism, it's all ok. But this lame example in no way "proves" that breaking standards is good. It's not and those poor users facing troubles trying to make it work, wouldn't be happy about these "shows". How do you think, would that HFS+ "ESP" be usable outside of iDevices? Given ESP should be kept anyway, it's not supposed to be easily destroyed - platform software may easily reject to destroy it, imagine you take off an HDD out of iDevice and want to use it with a PC, just installing there something else. And bang! No UEFI will understand that cr4p, your "wise" apple has put over there! Is this a good thing to show as an example of doing things right? well... so "objective".

Quote:
Octocontrabass wrote:
...Until you get a new SD card and start having problems.


In which case I just throw away that card, buy a different brand, and the rules of free market takes care of the rest ;-)

Cheers,
bzt

Aaand that market does use FAT in 99.9999% of cases leaving you with your stubbornness alone. and a lot of cards around you in your attempt to prove something. actually, it (the market), will be happy about that - because the stupid pays twice. ;)

_________________
future big goal: ANT - NT-like OS for mips, arm and x86.
current smaller goal: efify - UEFI for a couple of boards (mips and arm).


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Thu Jun 20, 2019 9:04 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 389
Your post was totally unnecessary.

zaval wrote:
Aaand that market does use FAT in 99.9999% of cases leaving you with your stubbornness alone. and a lot of cards around you in your attempt to prove something. actually, it (the market), will be happy about that - because the stupid pays twice. ;)
First, I'm not alone. There are plenty of SD card users using Linux filesystems (or even more extreme FSes) on boards like for example the Raspberry Pi. Tell them too that they are stubborn and alone! You can start with Elements 14. Just please post their answer here! :-)
Second, I have never ever found any SD card on the market which forces FAT, so no matter what you think or what you say, works for me :-D If I do eventually, I'll throw it away, I promise :-)

Make a deal with it, your efforts are futile no matter how aggressive you are, as you cannot convince me about something that goes against my every day experience. Just as nobody can convince me that 1 plus 1 is 3 (and believe me, they have tried, not kidding). I'm immune to brainwashing and marketing ;-)

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Fri Jun 21, 2019 3:20 am 
Offline
Member
Member

Joined: Wed Oct 30, 2013 1:57 pm
Posts: 304
Location: Germany
To reiterate on my point: Pointing to others and saying 'look, it works for me' does not cut any ice with people who point to the spec when talking about what a correct implementation looks like, because looking for what seems to work or not does not define what is compliant or not. All it really does is show that there are, at least, some people that are ignorant about what the spec dictates.

If you don't like the spec, go ahead and look for something else - but intentionally deviating from a spec which defines what hardware should look like and do just seems like a bad idea, mostly because you can't really change it (easily).

_________________
vineyard, a x86 OS.


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Fri Jun 21, 2019 5:26 am 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 485
bzt wrote:
your efforts are futile no matter how aggressive you are, as you cannot convince me about something that goes against my every day experience.
bzt

I think its worth noting that the belligerence in this entire thread started in your "Since when?" post. I knew this thread was headed nowhere good from that point, and here we are.


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Sat Jun 22, 2019 7:37 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 389
MichaelPetch wrote:
I think its worth noting that the belligerence in this entire thread started in your "Since when?" post. I knew this thread was headed nowhere good from that point, and here we are.
It is worth noting that "Since when?" was nothing more than a surprised and innocent question because I really do use ext4 on SD cards on daily basis for years now without any problems. And I'm just one among the many-many thousands of RPi users, and no one reported any problems with ext4 on SD, not a single case since RPi started many years ago.

This is a typical example of the lack of metacommunication in written posts, and when a question is read as offensive only because the reader wants to read it as offensive. Let's make this clear, nothing like that were meant.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Sat Jun 22, 2019 7:45 am 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 485
I didn't say the "Since When?" was belligerent. I said in your "Since When?" POST. That was to identify the post without actually having to repeat what you said. Re-read that entire post that started with "SInce When?" and any sane person would realize that the post was aggressive, pompous, and showed the same kind of arrogance and hostility that you used when discussing Brendan. This isn't the first time this has been a topic of concern in dealings with you. Personally I think you do it to be self aggrandizing and you may not even know you are doing it. In that post it was clear you were not going to accept any view contrary to yours. At that point it was going to be futile to have any level of rational discourse with you about what the spec actually says.


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Sun Jun 23, 2019 7:07 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 389
MichaelPetch wrote:
any sane person would realize that the post was aggressive
Why? Please elaborate. It can only be read aggressive if the reader is aggressive and wants to read it as such. I've just asked for proof, the spec namely (which as it turned out, Part1 the Physical Layer spec does not dictate FAT). There was no personal insults nor anything offensive in that particular post, just my most sincere surprise.

MichaelPetch wrote:
showed the same kind of arrogance and hostility
I'm just curious, if you call this calm and proof-based style of mine arrogant and hostile, then what would you call zaval's and Octrocontrabass' responses? The former can be read above, full of insults on the RPi community and accusations like I hate M$; the later insisted over and over again that my library does not do something that it wasn't supposed to do in the first place. How would you define that? Nice and friendly? (sarcasm, not offensive)

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Sun Jun 23, 2019 3:23 pm 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 396
Location: Ukraine, Bachmut
bzt, "Part1, the Physical Layer" is not the whole specification. It doesn't specify everything, other things are specified in other parts of the specification and are mandatory. when you ignore this fact, pointed several times to you by different people, in different style, you look a little bit strange. the part, that specifies the volume structure and file structure is called "SD Memory Card Specifications / Part 2. File System Specification Version 1.0". Version may vary, of course. Look it up on the web, and you'll find out the clear answer. like it or not, but this is how it is.

as of element14 and RPi. element14 reproduces the latter as it has been designed, it's not their design. They also make other open boards, like Beagle Bones. I do own one of such. These BB guys added eMMC on their boards, in addition to the SD slot - a wiser decision, so the OS (whateverOS) can live there with whateverFS it loves and SD card could be used with what it is meant to be used as FS (FAT) as removable storage - a much more suitable use case for the latter. eMMC is most often soldered on the main board, so it's much better to put OS there to not mess things up. Also, they are faster. SD cards as the target storage for the OS installation is not a good idea from the very beginning. SD cards were designed to serve the role of removable storage media. The only reason why there are SD card-only boards like RPi is cutting off on prices. Since linux can't live on FAT, they sacrifice compatibility, because they don't have choice. I told here already, when the topic hadn't degraded into these silly arguments with you, that on bare NAND storage devices, it's totally damaging for the NAND to put there ext4 or any other general purpose block oriented FS. Still, since the guys behind UBI/UBIFS gave up on supporting MLC NANDs, from now on, any new linux coming to such boards will smash that poor NAND mercilessly. But what's even worse, in most cases, all those linuxes have been coming with formatted ext4 without any supporting wear leveling layer beneath, even when UBI/UBIFS didn't discard MLC NANDs. Look at Cubieboards, Olinuxinos etc. they all come with ext4 on bare NAND. What does this prove? That they, whoever produce those images, don't care, nothing more. But per your logic, it's a proof of ext4 suitablity for bare NANDs, isn't it? *sarcasm
RPi also lets overclock SD cards to gain a little increase in the slow throughput - very bad move. Of course, you'd believe, this also wouldn't shorten cards' life span, since it's made by element14, *rolleyes but you know what - it does shorten cards' lifespan. because they aren't supposed to be used this way. They, who designed RPi, would be much better off to add support for UHS-I modes - that would increase throughput up to 3 times, lessen power consumption without exposing cards to any damage. But they haven't added this. Why? UHS-I is not something too new, too expensive. UHS-I cards are cheap and very ubiquitous for a long time. there are newer versions of cards - UHS-II, UHS-III and even freaking SDe - SD Express, using NVMHCI and being able to reach up to ~1GB/s of throughput (versus ~25MB/s of pre UHS-I, with 3.3 V signaling, your "brilliant" RPi could only use). They didn't care to add that. Despite most SD/MMC controller IPs, used in modern SoCs, are capable of UHS-I! So maybe they didn't care about incompliance with the SD specification as well? After all, a beardy community guru always can say - hey, your card was just a counterfeit sh1t, buy another one.
In sum - the specification, telling you that FAT is the part of it is not a proof for you, that FAT is the part of the specification. :shock: element14, that makes RPis the way they were designed is the proof that FAT is not the part of the specification. :mrgreen:

_________________
future big goal: ANT - NT-like OS for mips, arm and x86.
current smaller goal: efify - UEFI for a couple of boards (mips and arm).


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Sun Jun 23, 2019 8:41 pm 
Offline
Member
Member

Joined: Fri Aug 26, 2016 1:41 pm
Posts: 485
bzt wrote:
]I'm just curious, if you call this calm and proof-based style of mine arrogant and hostile, then what would you call zaval's and Octrocontrabass' responses? bzt
If you read back to my original post I said the problem STARTED with you. You then called out other people's aggressiveness and then I call you out for STARTING it. They responded to your hostility by having to be aggressive enough for you to look past your ego to actually try to tell you that you may be wrong. This was a civil convo until your s"Since When?" post that pretty much said you felt the people you were talking to were bullshitters despite the fact they apparently have read the spec and you didn't (Otherwise you would have been quoting the spec, and not asking them to do that for you).

Ultimately your view is that the spec be damned even though it has now been pointed out to you as to what it says. You are intent on simply falling back to your preconceived notions on the biases with total disregard to the fact that others may actually know more about this than you. As long as it works for the cases you use it in, it must be right. That's pretty much what you've said and you don't care what these people say or what the spec says.

I should point out that deflecting this onto Octo and Zaval who had to respond to your hostility is pretty much using the tu quoque logical fallacy, in particular whataboutism as a deflection away from the hostility you showed that originally sent this thread where it never had to go. This comment is my last on this subject.


Top
 Profile  
 
 Post subject: Re: Flash memory (SD) and File System Design Considerations
PostPosted: Thu Aug 15, 2019 2:06 am 
Offline
Member
Member
User avatar

Joined: Tue Oct 17, 2006 11:33 pm
Posts: 3875
Location: Eindhoven
Thread locked. Please create a new thread if you want to continue the Flash memory (SD) and filesystem design discussion, but please think of a useful and productive angle before doing so.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic This topic is locked, you cannot edit posts or make further replies.  [ 30 posts ]  Go to page Previous  1, 2

All times are UTC - 6 hours


Who is online

Users browsing this forum: JacobUnlot, Korona, MSN [Bot] and 14 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group