OSDev.org

The Place to Start for Operating System Developers
It is currently Sat Jan 19, 2019 2:54 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Fri Apr 20, 2012 3:15 pm 
Offline
Member
Member

Joined: Fri Nov 16, 2007 1:59 pm
Posts: 612
DavidCooper wrote:
The mismatch in disk size was imaginary - just a stupid error of mine caused by being new to hex editors.
I understood. I'm using HxD too.

Quote:
The BIOS reports 7936683 sectors by returning 254 127 243 for heads, sectors, cylinders (Int 13h AH=8); HxD shows 7897088 sectors now that I'm using it properly; Windows formats the drive to use 7887852 of them. ...
In CHS BIOS geometry SPT value can be up to 63, not more. Two high significant bits of this byte hold bits 8-9 of cylinder number. Function 8 is enough for my boot loaders (I use function 48h only in kernel to detect boot device by BIOS drive number). When "EDD instalation check" returns error for the drive you should use function 8 to get SPT and Heads parameters, and then should use them to translate LBA to CHS for function 2. Often there is a gap between end of partition and end of disk space because end of partition usually align up to end of cylinder for used CHS geometry (which can be different), or for reservation purposes. This gap usually is not used for flash drives. Don't use CHS geometry to calculate disk size, it's wrong (the gap between end of last whole cylinder and end of disk space is missed; returned number of cylinders can be not actual).

Quote:
I'm not looking for help debugging it just yet, and it's complex - it goes through tables of data loading them into registers and a DAP, then it calls Int 13h after loading AH with 2 or 42h depending on whether it's using CHS or LBA. The CHS version definitely works when loading from a floppy, and the values that end up in the DAP look correct. When attempting to boot from a flash drive, the code is definitely running through as it displays the CHS track number on the screen as it goes along (which doesn't have any impact on where it's actually reading from if it's using LBA), but it gets to a point where it stops loading and the beeper goes off repeatedly and I have to switch the thing off, so it isn't so much a crash as a freeze, though a very noisy one. I'll get it sorted eventually - it just takes so a long time setting up each experiment and my hands are falling to pieces due to RSI.
Evil in details.

_________________
If you have seen bad English in my words, tell me what's wrong, please.


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Fri Apr 20, 2012 4:28 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
Good news: it's just booted properly on the third attempt since my previous post, and I can load and save files and applications without trouble. I can now explore the machine more easily from the inside and use my OS to look around the flash drive that it's just loaded in from (which I couldn't do when booting from USB floppy because the BIOS can only handle a single USB device, that being the one it boots up from). It's important to understand why it wasn't working before, but I can see now that it was freezing with angry beeps when the location was wrong - the boot sector was loading empty sectors in and then running them. On the last two failed attempts it was reporting "Missing operating system" instead, but that was cured by replacing the original MBR code with my own - it seems that changing the 80h of the FAT32 partition's table entry to 0 and starting my partition's table entry with 80h wasn't enough to satisfy the existing MBR, but now that my code's replaced it and everything's lined up properly, it's all good. Looks like I won't need to write alternative CHS code at the moment after all (but I'll keep it on my to-do list as it'll doubtless still be needed on some machines, and certainly will for internal hard disks on ancient hardware).

Next step: put multiple copies of my OS in the special multi-floppy partition and set up a multi-boot system to select them. This may take a little while, but I'll release the latest version of my OS as soon as it's all working and is fully documented.

Next step after that will be to get hold of a few 16GB MicroSD cards and see if they work the way I hope, because I really need to be able to swap them over like floppy disks without upsetting the BIOS - the BIOS (this one at least - I don't know about others) cannot handle switching to another USB drive as it will only use the device it booted from (plus the machine's internal hard disk), but hopefully it won't object to a card being changed. If this doesn't work, the only way to make proper backups will be to do it through Windows and a hex editor, copying and pasting images.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Sat Apr 21, 2012 1:14 am 
Offline
Member
Member

Joined: Fri Nov 16, 2007 1:59 pm
Posts: 612
DavidCooper wrote:
the BIOS (this one at least - I don't know about others) cannot handle switching to another USB drive as it will only use the device it booted from (plus the machine's internal hard disk)
This is normal behaviour for the BIOSes.

Quote:
If this doesn't work, the only way to make proper backups will be to do it through Windows and a hex editor, copying and pasting images.
What does prevent you to do it under your OS? You can store images in different places of the disk space, and then can make choice between them during booting.

_________________
If you have seen bad English in my words, tell me what's wrong, please.


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Sat Apr 21, 2012 4:48 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
egos wrote:
What does prevent you to do it under your OS? You can store images in different places of the disk space, and then can make choice between them during booting.

Something could potentially go wrong and write to random sectors all through the flash drive, damaging every single copy of a file. It isn't at all likely, but it would be a mistake to think it couldn't happen, so backups have to be made to another device. Also, something in the device could break and make the entire content inaccessible at a stroke. If the MicroSD card idea doesn't work (I'm just about to try it out, and I'll be trying it with cards of different sizes), I might just have to work with flash drives, in which case I'll need to make backups within Windows every few days to prevent too big a loss if anything breaks, and that will involve copying and pasting data out using a hex editor (unless I export copies of everything into the FAT32 partition first as standard files - that would make it easier).

Whatever I do though, I will keep moving the latest version of the OS to new locations on the drive rather than repeatedly saving modified versions on top of the previous ones - I expect flash memory's getting more robust when it comes to how many times you can rewrite to it in the same place, but it still makes sense to try to maximise its lifetime by attempting to use the whole disk equally, and part of that involves not having any area of sectors that needs to modified everytime a file is created, changed or deleted, so the multiboot code (which will probably not all fit in the MBR) is going to need to hunt for the most recent versions every time by looking at the time and date stored in the many VBRs. When I have to jump back to using the start of the partition again, I'll move the subpartitions within it so that the directory of each floppy zone writes to an area which wasn't used so intensively on the first lap.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Sat Apr 21, 2012 8:32 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
I've done the experiment with a 2GB and 4GB MicroSD card and it failed - the BIOS objects to the card being changed and no longer recognises that drive, even if the first card is put back in, so it behaves in the same way as unplugging a USB drive and plugging in another one. I don't have two identical cards to try, but I suspect that that won't work either because it's probably reacting to some hidden code which identifies the individual card. The USB card reader also now fails to recognise the card I wrote my OS to and acts as if it's empty, though it still works with the other card, so there are other hazards to this too. I had to use an SD adapter instead to boot up the machine (via its SD card slot instead of using USB), so the experiment was done through that. I then restored the MicroSD card's original MBR and the USB card reader still refuses to talk to it.

berkus wrote:
Why making a direct disk image copies under windows or any other OS is a problem? You can dump entire flash drive into a file and upload it to a cloud storage, then to be re-downloaded and restored onto the media verbatim.

Why would I want to spend ages backing up a 16GB disk every time when only a few parts of it may have been changed since the last backup? Clearly it would be better to copy the changed files into the FAT32 partition and then back them up the easy way, though that involves taking the time to write code to do that, and I have higher priority work I'd rather be spending my time on. However, the real problem is that I'm dealing with program code and associated data files (relating to A.I.) which I don't want anyone else to be able to access. I don't want them on any machine that gets connected to the Web or on the hard disk of any machine that might need to go in for repair. I wanted to get everything onto MicroSD cards and to be able to keep it there without it ever going onto anything else (other than RAM). It now seems likely that that is not going to be possible.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Sat Apr 21, 2012 10:13 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
Ah, of course! I'm using machines with no internal optical drive, so I only normally see two available drives through the BIOS - the one I booted from and the internal hard drive. But what about that SD card slot which I booted from a few hours ago? Is it wired into the machine by a hidden USB link or is it more directly linked? Well, I've just booted from USB flash drive, but with a MicroSD card in SD card adapter in the SD card slot, and the result is exactly what I hoped it would be: three drives can now be accessed via the BIOS, so I can back up from flash drive to SD card, and I can then boot from SD to copy it onwards to another flash drive. This isn't as good as being able to switch between many MicroSD cards without having to reboot, but it'll do.

Now I can get back to my work.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Sun Apr 22, 2012 3:15 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
Quote:
Quote:
Why would I want to spend ages backing up a 16GB disk every time when only a few parts of it may have been changed since the last backup?

Because you can create a much smaller partition on the drive and copy only that? 4Mb would do for a start, you can always grow it by hand-editing the values in the MBR.

The things needing backed up will be spread over a much wider area than that from the outset, and copying out a partition that Windows doesn't recognise as a logical drive will never be the easiest approach either, but it doesn't matter any more as the SD card slot has come to the rescue: I'll be able to copy files from one device to another within my own file system and I'll only ever need to export partial disk images into Windows to make my OS available over the Web - it doesn't matter if that's slow as it'll be relatively rare action.

Anyway, thanks again to everyone who's helped me get this far with it - I'm finally free from the limitations of floppy disks (though I'm going to miss that chuntering noise).

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Mon Apr 23, 2012 8:06 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 09, 2010 8:57 am
Posts: 255
Location: Moscow, Russia
@DavidCooper,
I solve the backup task very simply. I perform the main development loop is in the Bochs. So I just prepare a simple and small enough files - HDD images which work in Bochs. The real hardware is primarily for test, - will it work on it too. The backup process is just a copying of HDD image file.

_________________
Yet Other Developer of Architecture.
OS Boot Tools.
Russian national OSDev forum.


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Mon Apr 23, 2012 4:11 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
berkus wrote:

Did - thanks for linking to that!

Yoda wrote:
@DavidCooper,
I solve the backup task very simply. I perform the main development loop is in the Bochs. So I just prepare a simple and small enough files - HDD images which work in Bochs. The real hardware is primarily for test, - will it work on it too. The backup process is just a copying of HDD image file.

I can see why many people would would find that the most convenient way of working, given that they're likely to be writing their OS code in a compiler or assembler under an operating system which may not be their own, so once they've got it converted into a disk image, the quickest way to test it is to boot it in an emulator, and it's also safer if it all goes horribly wrong. For me though, I do all my programming within my own OS - it became self-hosting at a very early stage in order to get away from loading via Qbasic (where it was originaly held as machine code in data statements). The way to achieve that was to have two copies of the first OS module which can modify and save each other. Ever since then I've been able to boot the machine and get to work within seconds instead of having to watch Windows waste several minutes of my life first, and then when I want to stop I can just save anything that needs saving and switch the thing straight off instead of having to wait a minute to see if it's going to hibernate or if it's stuck in a freeze. It also frees me from being pestered by idiotic messages from Windows about unused icons on my desktop while I'm trying to work on something so complex that it's taken me half an hour to load all the ideas involved in it into my head.

An emulator isn't the most comfortable working environment either as everything ends up in a small box instead of using the whole screen (unless you're doing something with a graphics mode the same size as the screen), there are irritating delays with everything, and particularly with cursor alignment when trying to get somewhere using a cursor key held down (although QEMU is a lot better than Bochs in that regard), the hardware scrolling doesn't work, the clock doesn't keep real time, the alarms are silent and the machine runs hot. I find emulators useful as a way of testing code that's likely to result in a crash or data corruption, and they're also useful as an indicator of things which might cause issues on some hardware, but my own OS running on real hardware is a much more comfortable programming environment, and it provides the added advantage of keeping all my program code and data off the internal hard disk (unless I actually want it to go there). I did write MBR code within QEMU though, since I was already going to be using Windows and HxD to write it to a flash drive.

Anyway, my plan is to go on working on real hardware, but instead of using a toppling pile of floppy disks I'm going to use two SD cards and two USB flash drives - I'll start by using one of the SD cards, keeping multiple copies of everything on that and occasionally backing up the most recent versions of things to one of the two flash drives. After a while, I'll stop using the SD card and switch to using the backup flash drive instead, making new backups from time to time on the second SD card. After another while, I'll start using that SD card as the device to work from and start using the second flash drive for backups. After another while, I'll use that flash drive to work from and bring the first SD card back into play for backups, and I'll just keep rotating through using those four devices in this manner. Additional backups of various things will be saved to other flash drives to keep a permanent record of the development of the OS, as well as of other software and files.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Tue Apr 24, 2012 4:46 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
Practicalities of using flash memory:-

It looks as if typical block sizes for flash memory (NAND, as used in SD cards and USB flash drives) are 16KB/128KB/256KB/512KB, meaning that when you write a sector, 32/256/512/1024 sectors may need to be collectively erased and rewritten. If using the BIOS to write to one of these devices and using many BIOS calls in the process, it isn't impossible that a large area of storage may be rewritten a hundred or more times, reducing the lifetime of the device considerably. Some of them appear to be able to handle a hundred thousand writes (according to Wikipedia), but that could easily be hacked down to well under a thousand multi-call saves via the BIOS because the same blocks may need to be rewritten so many times just to save a single file. Others devices can only handle a few thousand or even just a few hundred cycles. Wear leveling may reduce the impact of that on any particular memory location by switching blocks around behind the scenes (which would make it hard to test a device to see how many cycles it can actually handle), but it does look as if writing to flash memory in the same way as you would to a floppy disk needs to be avoided.

One way of reducing the amount of damage would be to rely on carefully aligned writes of 64 sectors every time such that a 512KB block would only need to be written a maximum of sixteen times per save (which should enable a 6000-rewrite lifespan for the more robust devices, while on some other devices you might only get 6 rewrites before it fails, although it shouldn't show up for a long time because of the wear leveling process that goes on behind the scenes). That could be achieved by taking each chunk of disk into memory to use like a RAM disk before writing the whole chunk back to disk in as few calls as possible. In my multi-floppy partition I intend to give exactly 1.5MB of space to each subpartition to keep the start of each aligned exactly with a block (that gives each one [edit]: slightly more space than would be needed to store an 85-track floppy disk [not exactly enough for 86 tracks as I originally stated here]), and each of these floppy-sized subpartitions can be treated as an independent disk, so it'll be easy to take one into RAM for use as a RAM disk before writing the whole thing back again at the end of a work session, thus allowing many files to be modified and saved without writing anything to flash memory until the very end of the process when they're all saved in one 48-call go (16 calls per block, assuming the largest block size which is the worst case scenario).

It may be that everything gets held in a cache within the device and isn't actually written until a certain amount of time has gone by, in which case multiple BIOS write calls could be combined into a single erase and write action, but it's safer not to assume that it works that way. The BIOS certainly can't leave it in a state where the data isn't written (or set to be written after a certain length of time) by the end of a call because it doesn't know if there will be more calls following. When you switch the machine off or remove the memory device, the data must actually be stored on it by that point as there appears to be no BIOS call designed to close the device before removing it. I'm going to assume that a block is written at the end of each BIOS call, and possibly two blocks if a group of sectors being saved by a single call crosses a boundary between blocks.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Last edited by DavidCooper on Wed Apr 25, 2012 2:11 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Wed Apr 25, 2012 7:36 am 
Offline
Member
Member
User avatar

Joined: Tue Mar 09, 2010 8:57 am
Posts: 255
Location: Moscow, Russia
@DavidCooper,
Don't worry about flash lifetime. Manufacturers did the best to rise it to unbelievable values.
1. Modern IC have already millions write cycles.
2. IC controller caches write requests and sequential requests will be recorded only once.
3. Internal rows have ECC codes to correct faulty bits.
4. Each IC has internal spare rows for that with unrecoverable errors. They allocate dynamically replacing bad ones on the fly.
In practice for many years I do all my work on Transcend USB stick. ICQ, mail, compilation projects, documents, web-design and many other stuff - all live there. And I never had faults. The only reason I upgraded it to another stick was the capacity and speed.

_________________
Yet Other Developer of Architecture.
OS Boot Tools.
Russian national OSDev forum.


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Wed Apr 25, 2012 2:06 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
If I was using my own device drivers to communicate directly with USB flash drives and SD cards I could use them the way they're actually designed to be used, but writing to them extensively via the BIOS must be pretty unusual, so it makes sense to take precautions to minimise the extra wear. If I need to do a lot of writing and rewriting of large files I should probably plug in a USB hard drive instead for that, but I can also take advantage of the space beyond the final partition of the internal hard drive to store cached copies of files which have been saved to a RAM disk but not yet to flash memory, thereby still enbling recovery of data if there's a crash - that's most important for files where I might be typing lots of stuff in and want to save them properly from time to time to avoid the risk of having to type it all in again. There's a spare 1.88MB available on the machine I usually program on, so that's more than enough for this purpose. On other machines there might be a lot less, but even just keeping a log of key presses would save repeating a lot of work in the event of a crash.

Because the SD card slot doesn't connect via USB, writing a device driver to communicate with it directly would be the easiest route to eliminating a large part of the problem, but I'm hoping the cards are robust enough not to need to spend time doing that in the near future - the odds are that they'll last long enough that I'll be replacing them with higher capacity ones before they're worn out anyway. I've decided to get slow cards as they're said to be able to take more cycles before failing. I've also decided to use the most compact USB flash drives I can find so that there's less danger of damaging them or the USB ports if I lean on them by mistake - they're going to be plugged in for many hours at a time, so the risk is higher than for using USB drives the more normal way. I've found some that only stick out about half a centimetre, so they're the best part of a centimetre shorter than the tiny MicroSD card reader I was originally planning to use.

Correction to an earlier post: 1.5MB is slightly less than would be needed for an 86-track floppy disk (some people claim to have managed to format 86 tracks, but it only works on some drives - 84 is more common).

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Wed Apr 25, 2012 2:18 pm 
Offline
Member
Member
User avatar

Joined: Wed Dec 01, 2010 3:41 am
Posts: 1761
Location: Hong Kong
DavidCooper wrote:
some people claim to have managed to format 86 tracks, but it only works on some drives - 84 is more common).


IIRC, I once format an 1.7MiB 3.5" floppy.


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Wed Apr 25, 2012 3:15 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 27, 2010 4:53 pm
Posts: 1001
Location: Scotland
bluemoon wrote:
DavidCooper wrote:
some people claim to have managed to format 86 tracks, but it only works on some drives - 84 is more common).

IIRC, I once format an 1.7MiB 3.5" floppy.

That would need 94 or 95 tracks, unless you were using a slow motor and packing more sectors into each track. The 84/86 track figures I mentioned may actually relate to CF2 floppy disks - it was a long time ago that I heard about it. If you could vary the motor speed (and maybe you have to if you're going to play tunes on them as in the youtube clip several posts ago), it should be possible to get well over 2MG on a floppy disk without losing any reliability on the inner tracks and without going beyond 80 tracks.

_________________
Help the people of Laos by liking - https://www.facebook.com/TheSBInitiative/?ref=py_c

MSB-OS: http://www.magicschoolbook.com/computing/os-project - direct machine code programming


Top
 Profile  
 
 Post subject: Re: Migrating OS from floppy to USB flash and USB hard drive
PostPosted: Thu Apr 26, 2012 1:00 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9269
Location: An der schönen blauen Donau, watching muppets
I once saw a reference to a trick to change physical sector size on floppies to reduce space spent on intersector gaps. They went on mixing sector sizes across the track to have large sectors where possible and smaller ones to fill up all the available space for maximum storage per sector.

I have no idea about the technical practicalities of that though. It's a bit of arcane magic you'll have to research yourself.

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 51 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: klange, Schol-R-LEA and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group