OSDev.org

The Place to Start for Operating System Developers
It is currently Mon Jul 23, 2018 6:01 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 13 posts ] 
Author Message
 Post subject: mtools (mcopy) acting weird?
PostPosted: Sun Jan 07, 2018 3:56 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
I thought I was going crazy but apparently I wasn't:
Every time I build my bootloader something crazy happens. There is a file called Stage_2.sys that gets copied on my virtual HDD using mcopy. So where is the problem?
"cluster_low" of that file(inside the root directory (of FAT 32)) increases by 1! So far it has reached 386!
What is going on, there is no logical explanation??? It is like the ONLY file on the entire HDD, there is free space everywhere, FAT is practically empty and mcopy decides to increment cluster_low by 1 each time and thus moving the entire file around...
Anybody has an idea of what could be going on? This is all of course happening on Linux.
Here are commands I use:
Code:
sudo mcopy -o Output/Stage_2.sys c:
# this is how c: is defined:
drive c: file="/pathTo/HDD.img" partition=1

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 2:17 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3206
Location: Chichester, UK
I suspect that this behaviour is by design. It increases the chances of recovery for deleted files as the data is not overwritten until there are no free clusters that haven't be used before. If you want the file to be written in the same place you need to format the disk before rewiting it.


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 5:32 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
iansjack wrote:
I suspect that this behaviour is by design. It increases the chances of recovery for deleted files as the data is not overwritten until there are no free clusters that haven't be used before. If you want the file to be written in the same place you need to format the disk before rewiting it.


Is there a way to prevent this permanently? I tired removing that file and then putting it back again, but doesn't work.
What do you mean by format the disk before rewriting it? Format in what way? It is already formatted as FAT 32.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 6:06 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3206
Location: Chichester, UK
No. I don't think you can alter this behaviour, assuming that it is part of the design of the FAT32 filesystem (other than writing your own handler for the filesystem). It shouldn't matter to a well-behaved program where a file is located on the disk.

By formatting the disk I mean that, if you want the file to occupy the same clusters on the disk you need to return it to exactly the same state as it was before you originally wrote the file. Deleting a file doesn't do this (for example, it leaves the directory entry in place with a marker indicating that the file has been deleted). So you need to format the disk again each time you want to write the file to that location. The real answer is that your program shouldn't depend on the absolute position of a file; it should parse the directory entry and file allocation table to determine the clusters used by the file.


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 6:46 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
iansjack wrote:
No. I don't think you can alter this behaviour, assuming that it is part of the design of the FAT32 filesystem (other than writing your own handler for the filesystem). It shouldn't matter to a well-behaved program where a file is located on the disk.

By formatting the disk I mean that, if you want the file to occupy the same clusters on the disk you need to return it to exactly the same state as it was before you originally wrote the file. Deleting a file doesn't do this (for example, it leaves the directory entry in place with a marker indicating that the file has been deleted). So you need to format the disk again each time you want to write the file to that location. The real answer is that your program shouldn't depend on the absolute position of a file; it should parse the directory entry and file allocation table to determine the clusters used by the file.


Even if I delete its directory entry the same thing happens. By design just doesn't sound correct. I took some time and had a look at other "real" FAT 32 formatted storage mediums. They all seem perfectly fine, FAT seems to be filled contiguously and first file has a cluster number of 3 (not 391 while having 390 free entries in front of it!). I am trying to make my bootloader as flexible as possible so there aren't much assumptions really, that is not a problem. What matters is that every time I press F5 (Build and Run) my file gets moved by 1 in the file allocation table leaving N - 1 free spaces behind it. Eventually FAT will collapse. Another thing I noticed is that FAT[3] contains F8 FF FF 0F (little endian) without having a file that corresponds to it.

And on last thing, after heavy experimenting and trial and error I managed to break everything :D
Code:
Fat problem while decoding 2 0
Streamcache allocation problem:C 2


Edit:
Adding F8 FF FF 0F back to FAT[3] somehow fixed to issue...
Don't know how it was related cause it wasn't linked to any file, it looks like it was put there by mkdosfs.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 7:20 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3206
Location: Chichester, UK
That FAT[3] entry is, I presume, for the root directory. Deleting it would certainly render the filesystem unusable.

Again, I emphasise, your programs should not rely on a file occupying a particular cluster. You need to obtain the position from its directory entry.

Note that the file information sector in FAT32 contains a field detailing the number of the last allocated cluster. You might consider what this field is used for.


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 7:52 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
iansjack wrote:
That FAT[3] entry is, I presume, for the root directory. Deleting it would certainly render the filesystem unusable.

Again, I emphasise, your programs should not rely on a file occupying a particular cluster. You need to obtain the position from its directory entry.

Note that the file information sector in FAT32 contains a field detailing the number of the last allocated cluster. You might consider what this field is used for.


Actually the root directory has its cluster number set to 0. While inside the BPB it is set to 2.
My code doesn't assume anything, it actually reads the cluster_low and cluster_high number and stores them for needed computation.

NICE CATCH: "FSInfo Sector" contains a field called "FSI_Nxt_Free", and guess what, it contains a value of 395. So that is where those numbers were coming from.
Looks like mcopy is to blame (kind of), it shouldn't change the starting cluster number when overwriting files. Looks like there isn't a way to fix this, even if I set "FSI_Nxt_Free" to 3 again.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 8:11 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 3206
Location: Chichester, UK
Octacone wrote:
Actually the root directory has its cluster number set to 0.
No. Cluster 0 isn't a valid data cluster and, in FAT32, the root directory is just another file.
Quote:
Looks like mcopy is to blame (kind of), it shouldn't change the starting cluster number when overwriting files. Looks like there isn't a way to fix this, even if I set "FSI_Nxt_Free" to 3 again.
I've already explained why FAT32 would write to unused clusters rather than previously used ones. It makes data recovery more likely. In addition, I would expect that it eases the task of finding a set of contiguous clusters large enough to hold a new file. Once all clusters have been used it will have to start reusing old ones - starting from the beginning again, these will be the clusters for the oldest deleted files.

If your code assumes nothing, why does this behaviour bother you?


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Mon Jan 08, 2018 12:02 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
insajack wrote:
Octacone wrote:
Actually the root directory has its cluster number set to 0.

No. Cluster 0 isn't a valid data cluster and, in FAT32, the root directory is just another file.


I am aware of that but:
Root_Directory_Cluster value inside the BPB = 2 (it is actually okay since arrays are zero based, forgot about that, I wasn't talking about FAT[3] but FAT[2])
Root_Directory_Entry itself cluster value = 0 (I think it should be zero actually, no clusters linked = 0 for standard directories (files) and 0 because it entry no.1)

insajack wrote:
Octacone wrote:
Looks like mcopy is to blame (kind of), it shouldn't change the starting cluster number when overwriting files. Looks like there isn't a way to fix this, even if I set "FSI_Nxt_Free" to 3 again.

I've already explained why FAT32 would write to unused clusters rather than previously used ones. It makes data recovery more likely. In addition, I would expect that it eases the task of finding a set of contiguous clusters large enough to hold a new file. Once all clusters have been used it will have to start reusing old ones - starting from the beginning again, these will be the clusters for the oldest deleted files.


Just tested this with an actual USB drive. I created an empty FAT 32 partition, put a file inside it, checked the value and it was 3, then I added another file and replaced the first one, 3 became 4 and newly create file had a value of 5. True!

insajack wrote:
If your code assumes nothing, why does this behaviour bother you?


That adds a whole new layer of complexity, like a lot. I guess I can handle the coding but there isn't really much space to do so. (420 bytes (VBR))

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Tue Jan 09, 2018 2:39 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 1280
Octacone wrote:
I guess I can handle the coding but there isn't really much space to do so. (420 bytes (VBR))

You could always use another sector for additional code, like how Microsoft does.


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Thu Jan 11, 2018 5:27 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
Octocontrabass wrote:
Octacone wrote:
I guess I can handle the coding but there isn't really much space to do so. (420 bytes (VBR))

You could always use another sector for additional code, like how Microsoft does.


Where am I allowed to put it? I know there are some reserved sectors, but Microsoft says they should be left unused for future expandability (that is likely to never happen in case of FAT).
I did notice some code in unusual places when inspecting a FAT 32 formatted USB doe.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Thu Jan 11, 2018 12:53 pm 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 1280
DOS-based Windows places the extra bootloader code in the third sector of the partition (LSN 2), right after the FSInfo sector. Like this. Since it's part of the boot record, there's a backup copy six sectors later (LSN 8).

Windows NT places the extra bootloader code in the thirteenth sector (LSN 12). Like this. I have no idea why they changed it.


Top
 Profile  
 
 Post subject: Re: mtools (mcopy) acting weird?
PostPosted: Fri Jan 12, 2018 1:10 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
Octocontrabass wrote:
DOS-based Windows places the extra bootloader code in the third sector of the partition (LSN 2), right after the FSInfo sector. Like this. Since it's part of the boot record, there's a backup copy six sectors later (LSN 8).

Windows NT places the extra bootloader code in the thirteenth sector (LSN 12). Like this. I have no idea why they changed it.


Thanks for the links! Knowing this will be very helpful.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group