Bootloaderis not consistent when booting in real hardware

[Octocontrabass]

If that were true, that would mean you couldn't boot anything else than FAT file systems, but I'm pretty sure you can boot Windows (NTFS) and Linux (ext2/3/4) on those machines just fine, so that cannot be the reason, it just happens your issue looks like it.

NTFS has a BPB with geometry in the same location as FAT, so even if the firmware patches the BPB, Windows is unaffected.

GRUB doesn't store anything where the BPB would go, so even if the firmware patches the BPB, Linux is unaffected.

Firmware only patches USB flash drives without a partition table. All other media, including USB flash drives with a partition table, are unaffected.

Nothing else uses it, and BIOSes are not checking for it (they actually can't check for it, that's just not possible).

They can check for and use it. I've found one Dell PC that uses the BPB to decide how INT 0x13 AH=0x02 should translate CHS to LBA. It hangs during POST if the BPB is mostly correct but specifies 0 heads per cylinder. (This only happens with unpartitioned drives, of course.)

If the intent is to write an MBR you wouldn't need a BPB. But you also wouldn't need to edit a partition table so I don't know what is being done at this time.

Some firmware will try to correct the geometry in the BPB when booting from an unpartitioned USB flash drive. The solution is to either have a BPB so the corrected geometry will end up where it belongs, or add a partition table with an active partition so firmware will detect that it's a MBR and stop trying to correct the nonexistent BPB.

That couldn't be true, as the first sector normally has the partition table, not the FS data.

USB flash drives don't always have a partition table. Sometimes the first sector of the disk is the first sector of the filesystem.

[liwinux]

Hold on a minute. Why are you writing to the partition? You're trying to install your code on the first sector of the disk, right?

I'm sorry my bad, I actually wanted to say that I'm writing right in the first sector and leave the partition as it is like no touching it. And apparently without a BPB, it doesn't boot nothing is displayed. So yeah I think the goal would be to write it.

Question: why are you testing this on real hardware? You should probably use an emulator until your ready for real hardware. Or at least be doing the real HW tests on a machine that isn't your primary machine.

I think you didn't red the whole thread. I know I could use QEMU, Virtualbox, etc .. But what if I want to test it on real hardware ? Like booting it directly from a USB drive ? Well that's exactly what I thought and while it is working fine with QEMU, it actually doesn't on real Hardware. So I was wondering why because I don't see what the issue could be. To get my point, every change that I make has to be tested out. Because imagine if I was writing my whole boot loader without testing it and just rely on emulation, and when I boot, I realize that nothing works ! Good luck to find out why It doesn't work ! The more code I write, the more I find it to be difficult to debug. Once I know that it's booting correctly then I will probably test it less often.

If I were you, I'd look for a different reason. Probably you need the 3 bytes jump at the beginning and the 2 bytes of magic at the end of the sector, but that's all.

Actually I have both of those thing but something is messing with code as for example, 2 of the 3 strings are printed or the beginning of my string is replaced with another character, so yeah I don't see any other possibility... I'm really confused like many think can cause this to happen.

[neon]

Hi,

2 of the 3 strings are printed or the beginning of my string is replaced with another character

I don't think you mentioned this detail before. This is a very strong indicator to the theory that the firmware may be modifying it.

This is basically what I meant for reserving space for a BPB (we use a fat32 BPB for this):

Code: Select all

bits 16
org 0x7c00
jmp 0x0000:BOOTSECT
times 0x5a db 0
;
; rest of code here...

If everything works perfectly then we can be sure that the firmware is for some reason thinking there is a BPB and trying to correct it. Keep in mind this is just a test - you can remove it afterwords. Just seeing if it'll work.

[liwinux]

If everything works perfectly then we can be sure that the firmware is for some reason thinking there is a BPB and trying to correct it. Keep in mind this is just a test

Wow, that was it, end of the story.. Honestly, without your help guys, I think I couldn't make it on my own. Now, is that OK if I just let this padding at the beginning ? I guess it's just a waste of memory after all...

But thank you all for your help, it finally worked as expected on both computers !

[nexos]

USB flash drives don't always have a partition table. Sometimes the first sector of the disk is the first sector of the filesystem.

That struck me after posting . Most USB flash drives still have a FAT FS, and BIOSes manufacturers care only about Windows.

[nullplan]

Also, ext2/3/4 have a 1024 byte offset from the start of the partition before they use any data, so the BIOS can overwrite as much as it wants in that first sector, it is not going to hurt the superblock.

[bzt]

Nothing else uses it, and BIOSes are not checking for it (they actually can't check for it, that's just not possible).

They can check for and use it.

Yeah, do they? And how? There's no magic bytes in the BPB, how do you check for the presence of a BPB? It's not possible not even in theory. You could use the strings "FAT16", "FAT32" etc., but even M$ says never do that, and there's no guarantee that an NTFS boot sector will have those at all.

I've found one Dell PC that uses the BPB to decide how INT 0x13 AH=0x02 should translate CHS to LBA. It hangs during POST if the BPB is mostly correct but specifies 0 heads per cylinder. (This only happens with unpartitioned drives, of course.)

I call this bullshit.

First, you say that Dell BIOS checks for the BPB, then you say it doesn't. Which one is it then? How does the BIOS know if the disk is unpartitioned (talking about non-GPT)?

Second, how do you distinguish an MBR from a VBR? They have exactly the same magic numbers! No reliable method exists to tell if there's a partitioning table or a BPB in a boot sector. Both MBR and VBR can start with a jump instruction and they both have the same magic bytes. As a matter of fact my boot loader uses exactly the same sector for both the MBR and VBR. (FYI all the disk tools are using the LBA address to decide, which method doesn't work in your case.)

It is more likely that you had a bad boot sector code that Dell BIOS refused to boot. And then when you put a BPB in it, you accidentally fixed the issue that was preventing the boot, but it wasn't for the BPB at all. Or when you've partitioned it, that override the entire boot sector replacing your bad code.

If the intent is to write an MBR you wouldn't need a BPB.

That's 100% true, therefore rumors about BIOS checking BPB is nothing more than myth.

Cheers,
bzt

[neon]

Hi,

Magic values is a part of the equation: you are supposed to read the bytes and check if they make sense and are valid values. If you limit the domain of boot records that you need to make sure will work it becomes a lot easier. Vendors are able to make assumptions about the operating systems that will be started - many of which may not be general rules. Other operating systems just have to adopt if they want to run on these machines.

You are looking for a general solution to a problem that only exists in practice.

[Octocontrabass]

Yeah, do they? And how? There's no magic bytes in the BPB, how do you check for the presence of a BPB? It's not possible not even in theory. You could use the strings "FAT16", "FAT32" etc., but even M$ says never do that, and there's no guarantee that an NTFS boot sector will have those at all.

I don't think I ever figured out exactly what the Dell BIOS was looking for when deciding whether a disk contained a BPB or not. I do know that it wasn't very good: if I formatted an unpartitioned flash drive as FAT and overwrote offset 0xD (sectors per track) with 0, it decided no BPB was present, but doing the same with offset 0xF (heads) caused it to hang during POST.

First, you say that Dell BIOS checks for the BPB, then you say it doesn't. Which one is it then?

Where did I say it doesn't?

How does the BIOS know if the disk is unpartitioned (talking about non-GPT)?

Second, how do you distinguish an MBR from a VBR?

That depends on the firmware. Some will look for a valid BPB, and if they don't find one, they assume the disk is partitioned. Some will look for a valid partition table, and if they don't find one, they assume the disk is unpartitioned. Some will look for both a valid BPB and a valid partition table, and if they find neither, they refuse to boot the disk. Of course, there is no standard for what counts as "valid".

It is more likely that you had a bad boot sector code that Dell BIOS refused to boot. And then when you put a BPB in it, you accidentally fixed the issue that was preventing the boot, but it wasn't for the BPB at all. Or when you've partitioned it, that override the entire boot sector replacing your bad code.

No, I assure you, it would hang during POST. It shows a loading bar on the screen during POST. If I used Windows to format an unpartitioned flash drive to FAT and changed the byte at offset 0xF to 0, the loading bar would stop partway.

[bzt]

doing the same with offset 0xF (heads) caused it to hang during POST.

Sounds to me like a BIOS bug. I don't think it worth the effort to support that particular buggy BIOS.

First, you say that Dell BIOS checks for the BPB, then you say it doesn't. Which one is it then?

Where did I say it doesn't?

Here:

(This only happens with unpartitioned drives, of course.)

That's why I asked how does the BIOS know if the disk is unpartitioned?

How does the BIOS know if the disk is unpartitioned (talking about non-GPT)?

Second, how do you distinguish an MBR from a VBR?

That depends on the firmware. Some will look for a valid BPB, and if they don't find one, they assume the disk is partitioned.

Hold on right there. You said BPB is only checked if the disk is unpartitioned, therefore you cannot use the BPB check to determine if the disk is partitioned.

Some will look for a valid partition table

But how? Can you explain how to detect for an MBR partitioning table existence? (Other than it is in the first sector).
I mean, can you give an implementation for this?

Code: Select all

/* return 1 if there's a partitioning table included, 0 otherwise */
int hasPartitioningTable(char sector[512])
{
    /* I'm curious about this part */
}

But take a step back, why is there a need for any BIOS to know if there's a partition on the disk? According to the BIOS Boot Specification, the BIOS just loads the first sector, it doesn't care nor need to know if there are partitions or not. Any BIOS which does, is not BBS compliant and should be considered buggy and faulty.

Some will look for both a valid BPB and a valid partition table, and if they find neither, they refuse to boot the disk. Of course, there is no standard for what counts as "valid".

As I've said before it simply does not worth the effort to support such buggy BIOSes which violate the BBS specification. They are extremely rare, and now with UEFI taking over on real machines and BIOS soon will be exclusive to VMs it simply doesn't matter any more. (Plus this only happens if you choose "floppy emulation for USB" in the BIOS setup, so the solution on your Dell is just to uncheck that option. Sometimes called "use HDD emulation for USB" with reverse flag, but that's the same.)

The best and simplest solution is:
1. start your sector with a jump instruction (E8h or E9h)
2. finish it with 55h AAh magic
3. leave both the BPB and partitioning table area empty (leave it to be filled up by another tool, like fdisk or mkfs.vfat for example)
4. play with your BIOS setup, there'll be a configuration which does not check for BPB nor partitions

My boot loader does this, and it works as MBR, VBR and even as CDROM "no emulation" boot record too, and nobody ever complained about any BIOS refusing to boot it on real hardware, regardless to the configuration. It Just Works (tm)
BTW, if you still have that old Dell, I would appreciate if you could give a try to boot BOOTBOOT on it and see if it works.

Cheers,
bzt

[Octocontrabass]

Hold on right there. You said BPB is only checked if the disk is unpartitioned, therefore you cannot use the BPB check to determine if the disk is partitioned.

No, I said this particular Dell BIOS only uses the CHS geometry from the BPB for INT 0x13 AH=0x02 when the disk is unpartitioned. It may still examine the BPB when determining whether the disk is partitioned or not.

But how? Can you explain how to detect for an MBR partitioning table existence?

It's different for each BIOS I've examined. Some look for exactly one "active" partition and assume any values other than 0 or 0x80 means no partition table is present. Some examine the CHS or LBA addresses to see if the partitions physically fit on the disk and don't overlap each other or the MBR. There are probably other methods in use, these are just the ones I was able to confirm through testing.

But take a step back, why is there a need for any BIOS to know if there's a partition on the disk?

Because it maintains compatibility with DOS without forcing the user to change BIOS settings.

play with your BIOS setup, there'll be a configuration which does not check for BPB nor partitions

On most of the PCs I've tested, there is no such setting.

BTW, if you still have that old Dell, I would appreciate if you could give a try to boot BOOTBOOT on it and see if it works.

I don't have room to set it up at the moment. I expect it'll work, unless you've intentionally crafted a BPB that makes it hang.