OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 2:31 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
 Post subject: Re: Using FUSE as an VFS
PostPosted: Thu Mar 25, 2021 10:00 am 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 401
bzt wrote:
Octocontrabass wrote:
rdos wrote:
Still, is there any reason to support exFat at all?
It's mandatory for SDXC, and cards may misbehave if you don't format them according to the SD standard.
WARNING: fact check: FAKE
Oh, not this bullshit again, please. SDXC cards are perfectly capable of storing other filesystems, nobody ever reported a problem, never. Plus even the SD standard defines a register to tell that the card isn't using exFAT, which register must be supported by all compliant drivers. So no, not mandatory for several reasons, period.

Cheers,
bzt



Not sure it's fake. Misbehave doesn't mean fail. It might be that the SDXC card has optimisations that work with exFAT, and not using exFAT bypasses those optimisations, which might result in poor performance or excessive wear. For example, some cards have been shown to have FTL optimisations so the FAT area of the disk is optimised for small writes, with more fine grained FTL mapping in those areas, whereas general data areas tend to have coarser grained FTL.

So while they're perfectly capable of storing other filesystems, it is absolutely possible that doing so is not optimal.


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Thu Mar 25, 2021 1:24 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5099
bzt wrote:
No, their reasoning is invalid, because when you're checking file names in a case-insensitive way, you must convert all i variants (İ i I ı) to the same uppercase dotless I.

Do you have a source for this claim? Everything I can find says İ/i and I/ı are considered two separate and distinct letters in Turkish. In addition, the up-case table recommended in the exFAT specification doesn't provide a conversion to uppercase for ı.

bzt wrote:
Actually the current solution introduces a pretty nasty bug. Let's assume you have an application that wants to open a filename with an İ in it from an exFAT formatted image. If you compile that and create the image on a machine with user's language set to Turkish, it will work, but in every other case the app will return "File not found".

Why would it do that? Maybe I'm missing something here, but I don't see any reason why changing the user's language would suddenly make it impossible to open files with İ in the name.

bzt wrote:
Plus even the SD standard defines a register to tell that the card isn't using exFAT, which register must be supported by all compliant drivers.

I seem to have misplaced my copy of the SDXC specification. Can you quote the relevant paragraphs?


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Thu Mar 25, 2021 2:36 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
bzt wrote:
Plus even the SD standard defines a register to tell that the card isn't using exFAT, which register must be supported by all compliant drivers.

If only everyone followed the standard... I'm sure many drivers assume exFAT. With that being said, you are right for the most part, as Windows allows you to format SDXC cards as FAT32,

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Fri Mar 26, 2021 2:04 pm 
Offline
Member
Member

Joined: Wed Oct 01, 2008 1:55 pm
Posts: 3191
I have most of the disc buffering interface done now, and it occurred to me that instead of letting the filesystem block on IO when it would read data, it can instead create a request for x sectors, link these sectors to a request structure, and as soon as those are done the request is marked as done and the owner thread is signalled. This way, the filesystem driver would mostly handle meta data and create queued requests for file data. Maybe this is good enough so there is no need to create parallell threads in "fuse".

I have 16 bits available per sector buffer, and I initially thought that I'd create a linked list of selectors, but it might be better to use it as a bitmap instead. One bit can be for general IO and the 15 other bits for partitions. Since partitions have defined areas, which paritition a sector belongs to is rather easy to calculate, and then the 15 bits would point to one of 15 possible active requests in the partition.

So, when a partition wants to request 100 sectors of file data, it picks one of the 15 request structures, fills in the termination info, creates requests for the sectors and sets the bitmap bit based on the request # in each of them. The request structure will have a counter of currently pending sectors, and every time a sector is finished, the counter is decreased in active requests and when the counter goes to zero, the owner thread is signalled.

Bit 0 will be special. It would be used to read/write raw sector data (either from initialization code or applications). When a thread requests a sector, a request structure is dynamically allocated, the sector # is filled in, as well as the thread to notify on completion. This structure is then placed in a link list, and bit 0 in the bitmap is set in the sector buffer.


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Fri Mar 26, 2021 4:11 pm 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 401
rdos wrote:
I have most of the disc buffering interface done now, and it occurred to me that instead of letting the filesystem block on IO when it would read data, it can instead create a request for x sectors, link these sectors to a request structure, and as soon as those are done the request is marked as done and the owner thread is signalled. This way, the filesystem driver would mostly handle meta data and create queued requests for file data. Maybe this is good enough so there is no need to create parallell threads in "fuse".


I was literally thinking the same thing today, in which case the filesystem just becomes a complex device mapper.

I was thinking how to handle data transfer from device to user space with minimal copying, and basically the solution you describe above seemed to be the answer.

But that would only work for block based file systems, and so would be no use to the likes of sshfs or similar filesystems, which aren't backed by a block device, so you'd still need a buffer mechanism to pass data to/from the FUSE module, in which case a shared mapping might be a better strategy for copy avoidance.


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sat Mar 27, 2021 1:07 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
Okay, SD card one last time, here we go.

thewrongchristian wrote:
Not sure it's fake. Misbehave doesn't mean fail. It might be that the SDXC card has optimisations that work with exFAT, and not using exFAT bypasses those optimisations
First, "misbehave" suggests malfunctioning or corrupt filesystem, while "bypassing optimization" suggests it's slow, but correct. (Neither happens BTW)

thewrongchristian wrote:
which might result in poor performance or excessive wear. For example, some cards have been shown to have FTL optimisations so the FAT area of the disk is optimised for small writes, with more fine grained FTL mapping in those areas, whereas general data areas tend to have coarser grained FTL.
Second, no there's absolutely no performance drawback, nor any other drawback, because it is not the card that classifies the sectors, but the host (see simplified SD card specification section 4.13.2.6 Distinction of Data Type and section 4.13.2.8 Speed Class Control). The host is supposed to tell the controller if it's writing directory data, allocation data or file data. A high speed write consist of: "The Speed Class specification for SDXC defines the CI Update sequence and operation time. The sequence is similar to the directory entry update. CI is always written by a 512-byte single block write (either CMD24 or CMD25) and preceded by CMD20 Update CI command. "

An ext4 (or any other) driver for example can easily comply with that, use these commands in this order, while providing its own (non-fat) data structures in the sectors.

Octocontrabass wrote:
I seem to have misplaced my copy of the SDXC specification. Can you quote the relevant paragraphs?
Nope, you just don't want to read. Even spoken of in the "simplified" spec. I've already did quote it, several times actually... In the simplified SD specification
Code:
4.13.2.7.3 Requirements of SD File System
This specification can be applied only to the SD file system formatted card defined by the File System Specification Version 3.00.
Pay good attention on the wording: it does not say other file system's can't be used, it says "the following is true if exFAT is being used".

The simplified SD specification does not go into details, but it names that register specifically: see section 5.3.2 CSD Register, bits FILE_FORMAT_GRP and FILE_FORMAT bits.
And on page 12, in Table 3-2 : SD Memory Card Registers, it says:
Quote:
CSD 128 Card Specific Data; information about the card operation conditions(See 5.3). Mandatory
Note that last bold word. Supporting CSD bits (including FILE_FORMAT bits) is mandatory. According to this, SDXC cards are allowed to use just as any file systems they want, exFAT is just optional (but pushed by M$ marketing department aggressively because they have active patents on it and they are making lots of money from the licensing fees.)

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sat Mar 27, 2021 1:31 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
Octocontrabass wrote:
Do you have a source for this claim? Everything I can find says İ/i and I/ı are considered two separate and distinct letters in Turkish.
You've answered your own question: "considered separate in Turkish". And only in Turkish may I add, so if my machine uses any other localization, those aren't separate.

Octocontrabass wrote:
bzt wrote:
Actually the current solution introduces a pretty nasty bug. Let's assume you have an application that wants to open a filename with an İ in it from an exFAT formatted image. If you compile that and create the image on a machine with user's language set to Turkish, it will work, but in every other case the app will return "File not found".

Why would it do that? Maybe I'm missing something here, but I don't see any reason why changing the user's language would suddenly make it impossible to open files
Isn't that obvious? There can be only two possibilities:
a) If the UNICODE upper-case table is the same for all localizations, then there's absolutely no point in saving it on each and every disk.
b) And if it's not the same, differs per localizations, then exFAT file systems aren't portable, because a file saved on a Turkish machine might have a different upper-case name than a file saved on an English machine, hence causing file name collisions or file name not found errors depending on the user's language.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sat Mar 27, 2021 6:59 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5099
bzt wrote:
An ext4 (or any other) driver for example can easily comply with that, use these commands in this order, while providing its own (non-fat) data structures in the sectors.

Does that work even with USB SD card adapters? Last I checked, those don't give you any control over which commands are sent to the SD card.

bzt wrote:
You've answered your own question: "considered separate in Turkish". And only in Turkish may I add, so if my machine uses any other localization, those aren't separate.

My machine is not localized for Turkish, yet I can save a file that's named "i" and it's considered separate from a file named "ı". Or, I can save a file named "I" and it's considered separate from a file named "İ". (Also, Turkish is not the only language where İ/i and I/ı are separate letters.)

bzt wrote:
b) And if it's not the same, differs per localizations, then exFAT file systems aren't portable, because a file saved on a Turkish machine might have a different upper-case name than a file saved on an English machine, hence causing file name collisions or file name not found errors depending on the user's language.

But in your example, the file name contains "İ" and the program trying to open the file is searching for a file named "İ". Even if converting the name to uppercase changes "İ" into something else, the names will still match because both will be converted using the same up-case table.

Name collisions don't happen until you start trying to move files between the exFAT disk and something else.


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sun Mar 28, 2021 1:12 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
Octocontrabass wrote:
bzt wrote:
An ext4 (or any other) driver for example can easily comply with that, use these commands in this order, while providing its own (non-fat) data structures in the sectors.
Does that work even with USB SD card adapters?
Absolutely. I do that on a daily basis. Moreover, I don't use a single file system, I use multiple with a partitioning table. I transfer files to my Raspi and my PC using an USB adapter, and all the cards are formatted with MBR+FAT+ext4, as RaspiOS likes its boot partition as FAT32 and root partition as ext4 (neither being exFAT). Nobody (hundred thousands RaspiOS users all around the world) have reported issues with that.

Octocontrabass wrote:
Last I checked, those don't give you any control over which commands are sent to the SD card.
It doesn't matter if USB SD adapter doesn't give you control over commands, because it's still the host that classifies the sectors and not the card. Worst case scenario, your USB adapter driver is using simple sector read / writes instead of the high speed CMD20 command, so it's going to be slower, but perfectly working. I did not noticed lower performance with my Kingston USB adapter BTW.

Octocontrabass wrote:
My machine is not localized for Turkish, yet I can save a file that's named "i" and it's considered separate from a file named "ı".
Sure, there's no issue with one computer. The problem appears when you move that file system between two computers with different locales, which need different upper-case for the same UNICODE code-point.

Octocontrabass wrote:
the names will still match because both will be converted using the same up-case table.
That's the point, not the same up-case table. What if one machine creates an upper-case table with a filename, and another machine wants to create another file with a different upper-case, but it can't because according to its upper-case that would map to the same name? Should it overwrite the table and create a new file? Or will it silently overwrite the existing file? What if it overwrites the table, creates a new file, and the file system is moved back to the original machine?

To very simply put, machine A wants an upper-case table where names X and Y would be the same. Machine B wants an upper-case table where names X and Y are different. Obviously machine A and machine B can't save both their upper-case tables to the same file system, because there's only one table. Such a file system will fail to work either on machine A or on machine B.

Octocontrabass wrote:
Name collisions don't happen until you start trying to move files between the exFAT disk and something else.
Yes, it could happen. That's because upper-case translation is not bijective as it differs per localization. That's why you shouldn't store localized names, a file system must store names in a localization independent format, and only apply the localization in the driver according to the current locale so that the file system can be used on different machines.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sun Mar 28, 2021 3:23 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5099
bzt wrote:
Nobody (hundred thousands RaspiOS users all around the world) have reported issues with that.

Didn't zaval report having exactly this kind of issue? I'm sure I've seen at least one other report of an SD card corrupting files only when formatted with ext4, but I can't remember where I found it or what I was searching for.

bzt wrote:
That's the point, not the same up-case table.

Why not? The VFS should use the up-case table on the volume when accessing that volume.

bzt wrote:
What if one machine creates an upper-case table with a filename, and another machine wants to create another file with a different upper-case, but it can't because according to its upper-case that would map to the same name? Should it overwrite the table and create a new file? Or will it silently overwrite the existing file? What if it overwrites the table, creates a new file, and the file system is moved back to the original machine?

There is only one correct behavior: tell the user a file already exists with that name.

bzt wrote:
To very simply put, machine A wants an upper-case table where names X and Y would be the same. Machine B wants an upper-case table where names X and Y are different. Obviously machine A and machine B can't save both their upper-case tables to the same file system, because there's only one table. Such a file system will fail to work either on machine A or on machine B.

Or both machines will use the up-case table on the volume and behave the same way, even when this behavior might be inconsistent with the user's locale. User A might be confused if names X and Y are allowed to coexist in the same directory on an exFAT volume when they're not allowed to coexist elsewhere. User B might be confused if names X and Y are not allowed to coexist in the same directory on an exFAT volume when they're allowed to coexist elsewhere.

Consider the alternative: user B creates files X and Y, then user A tries to open one of them. On machine A, both files have the same name, so which file gets opened? I suspect the up-case table exists specifically to prevent this scenario.

bzt wrote:
That's why you shouldn't store localized names, a file system must store names in a localization independent format, and only apply the localization in the driver according to the current locale so that the file system can be used on different machines.

What does this even mean? Names are always localized!


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sun Mar 28, 2021 5:30 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
Octocontrabass wrote:
bzt wrote:
Nobody (hundred thousands RaspiOS users all around the world) have reported issues with that.

Didn't zaval report having exactly this kind of issue? I'm sure I've seen at least one other report of an SD card corrupting files only when formatted with ext4, but I can't remember where I found it or what I was searching for.
Nope. Plus if only exFAT were supported, you couldn't use partitioning on the SD cards, but everybody can.

Octocontrabass wrote:
bzt wrote:
That's the point, not the same up-case table.

Why not? The VFS should use the up-case table on the volume when accessing that volume.
You still don't get it do you? Let me put it as simple as possible: Turkish machine would put Turkish table on the volume, English machine would put English table. Turkish table != English table.

Octocontrabass wrote:
There is only one correct behavior: tell the user a file already exists with that name.
So you admit, there can be a name collision.

Octocontrabass wrote:
Or both machines will use the up-case table on the volume and behave the same way
Nope. For the last time, I'll try to explain.

Let's say given application A, which operates on a removable device. A is installed on two (otherwise completely identical) computers, a Turkish and on an English one. Depending if the removable media used by the application was formatted on the Turkish machine or on the English machine, exactly the same A application on the Turkish machine would behave differently than exactly the same A application on the English machine. All it takes to trigger the bug is to use a removable media formatted on the other machine (hence using different tables).

Octocontrabass wrote:
What does this even mean? Names are always localized!
You are confusing name localization with name encoding, just like those M$ guys who created this terrible exFAT specification. You're wrong. UTF-8 encoding for example isn't localization dependent. It works exactly the same way on a Turkish machine and on an English machine. Filenames are encoded and stored to the file system exactly the same way, bit-by-bit and decoded on the other machine correctly no matter the user's locale setting. Upper-case matching should be implemented in the driver, independently to the file system, so both the Turkish machine and the English machine would display correctly the name in uppercase. Then there would be no issues when the media is moved from one machine to another, because the file system would only contain localization-independent encoding of the file names, no matter if it was formatted on a Turkish machine or on an English machine.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Sun Mar 28, 2021 2:48 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5099
bzt wrote:
You still don't get it do you? Let me put it as simple as possible: Turkish machine would put Turkish table on the volume, English machine would put English table. Turkish table != English table.

Correct.

bzt wrote:
So you admit, there can be a name collision.

Yes, of course. You must be prepared for this anyway, since users may try to give the same name to two different files.

bzt wrote:
Let's say given application A, which operates on a removable device. A is installed on two (otherwise completely identical) computers, a Turkish and on an English one. Depending if the removable media used by the application was formatted on the Turkish machine or on the English machine, exactly the same A application on the Turkish machine would behave differently than exactly the same A application on the English machine. All it takes to trigger the bug is to use a removable media formatted on the other machine (hence using different tables).

Again, correct.

bzt wrote:
You are confusing name localization with name encoding, just like those M$ guys who created this terrible exFAT specification. You're wrong. UTF-8 encoding for example isn't localization dependent. It works exactly the same way on a Turkish machine and on an English machine. Filenames are encoded and stored to the file system exactly the same way, bit-by-bit and decoded on the other machine correctly no matter the user's locale setting.

I think you are confused here. exFAT stores file names in UTF-16, and they are encoded and decoded exactly the same on all machines regardless of the user's locale.

bzt wrote:
Upper-case matching should be implemented in the driver, independently to the file system, so both the Turkish machine and the English machine would display correctly the name in uppercase.

The uppercase file name is used only for case-insensitive file name comparison. It is never stored on the disk or displayed to the user.

bzt wrote:
Then there would be no issues when the media is moved from one machine to another, because the file system would only contain localization-independent encoding of the file names, no matter if it was formatted on a Turkish machine or on an English machine.

How does this resolve the problem when two files on the disk have different names in one locale and the same name in another locale? When two files have the same name, how does the user choose which one to access?


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Mon Mar 29, 2021 2:26 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
Octocontrabass wrote:
I think you are confused here. exFAT stores file names in UTF-16, and they are encoded and decoded exactly the same on all machines regardless of the user's locale.
That's where you're mistaken. The conversion table is stored in the file system, and it does depend on the user's locale. Meaning you'll get a different exFAT image if you format it on a Turkish machine and if you format it on an English machine.

Octocontrabass wrote:
The uppercase file name is used only for case-insensitive file name comparison. It is never stored on the disk or displayed to the user.
That's the problem, it is stored on disk, but not shown to the user. You can't see if a medium was formatted on a Turkish machine or on an English machine, yet it influences file names and therefore application behaviour greatly.

Octocontrabass wrote:
How does this resolve the problem when two files on the disk have different names in one locale and the same name in another locale?
It does not solve that, it makes the behaviour deterministic and predictable, consistent without hidden traps. For machines with locale A would always be different, for machines with locale B would always be the same, regardless in which machine the medium was formatted.

Cheers,
bzt


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Mon Mar 29, 2021 2:44 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5099
bzt wrote:
That's where you're mistaken. The conversion table is stored in the file system, and it does depend on the user's locale. Meaning you'll get a different exFAT image if you format it on a Turkish machine and if you format it on an English machine.

The conversion table is not part of the file name. The file names are stored exactly the same.

bzt wrote:
That's the problem, it is stored on disk, but not shown to the user. You can't see if a medium was formatted on a Turkish machine or on an English machine, yet it influences file names and therefore application behaviour greatly.

You're right, it's not a great solution, but it's still better than allowing two files with the same name.

bzt wrote:
It does not solve that, it makes the behaviour deterministic and predictable, consistent without hidden traps. For machines with locale A would always be different, for machines with locale B would always be the same, regardless in which machine the medium was formatted.

Two files are created in locale A. In locale B, the file names are equal. What happens when the user of machine B tries to open one of the files? Do they get the file they chose? Remember, both files have the same name! Machine B can't tell them apart!


Top
 Profile  
 
 Post subject: Re: Using FUSE as an VFS
PostPosted: Mon Mar 29, 2021 3:49 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 1584
Octocontrabass wrote:
bzt wrote:
It does not solve that, it makes the behaviour deterministic and predictable, consistent without hidden traps. For machines with locale A would always be different, for machines with locale B would always be the same, regardless in which machine the medium was formatted.
Two files are created in locale A. In locale B, the file names are equal. What happens when the user of machine B tries to open one of the files? Do they get the file they chose? Remember, both files have the same name! Machine B can't tell them apart!
Exactly the same thing like now. The difference is, it would only depend on the application's user locale (which the user can easy change if they experience problems), and not on some hidden property of the file system they can't see nor change.

(And let's not get into what happens if a medium is formatted on locale A, and then moved to machine with locale B which supposed to save a different conversion table. Will locale B's conversion be ignored? Or will it be merged with the table on the file system? Can you create a uppercase table with both Turkish and Devanagari uppercase conversions for example? Or is it either this or that? In theory it could be stored, but do drivers support that? Should they? So many usability questions with this custom table on disk solution...)

Cheers,
bzt


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 23 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group