Initramdisk is corrupted

[Compaszer]

Hello.
I am writing a monolithic x86 operating system in rust.
After I included some files via the include_bytes! macro in my kernel, the initramdisk appears to be corrupted on real hardware, but not within Qemu. I've checked this at the rust entry point of my os before any actual kernel code gets executed, so allocator or paging can't be the problem. So I am not really sure what could cause this problem. It must be related to the increased kernel size, but I assume grub2 to place everything correctly and non overlapping.
I would be glad for any kind of help.

[Octocontrabass]

the initramdisk appears to be corrupted

How so?

[ZeSystem]

"Welcome to the classic OSDev rite of passage: 'It works in QEMU, but breaks on real hardware.' Since this happens at the very entry point before paging or your allocator is active, the issue is almost certainly related to how GRUB maps the kernel and the initrd into physical memory based on your ELF headers and the machine's actual memory map.

Here are the most likely culprits you should investigate:

1. E820 Memory Map Overlap (Highly Likely)
QEMU maps RAM flatly and continuously. Real hardware has "holes"—memory regions reserved for the BIOS, ACPI tables, or MMIO. When you included those bytes, your kernel grew, pushing the initrd further up in physical memory. It's very likely that GRUB is now loading the initrd into a reserved E820 region. The hardware might ignore writes to it, or firmware might overwrite it, resulting in corruption.

2. Module Alignment Issues
By default, GRUB doesn't necessarily align modules to page boundaries unless explicitly told to. Your increased kernel size shifted the offset where the initrd begins. If it now lands on an unaligned physical address, certain real-hardware memory controllers or CPU cache behaviors can cause alignment faults or corrupt reads when accessed as words.

Fix: Ensure your Multiboot header has the alignment flag set (e.g., bit 0 MULTIBOOT_PAGE_ALIGN for Multiboot 1).

3. Linker Script vs. BSS Overlap
include_bytes! places data directly into the .rodata section, expanding your ELF's loadable segments. Check your linker script. If GRUB miscalculates the total size of the kernel (or if your .bss section is placed incorrectly relative to .rodata), GRUB might load the initrd inside your kernel's actual memory space, or right where the .bss section is supposed to be zeroed out later.

What to check next:
Could you modify your early Rust entry point (using direct VGA text mode writing or early serial logging if you have it) to print the hex values of:

initrd_start

initrd_end

Your kernel's _end symbol (from the linker script)

Also, run readelf -l your_kernel and check the LOAD segments to ensure MemSiz and PhysAddr look correct after the asset inflation. What do the physical addresses look like on real hardware vs QEMU?"

[Compaszer]

the initramdisk appears to be corrupted

How so?

My tar reader failed to read it, because Strings suddenly contained invalid characters. So I've calculated a simple checksum of the initramdisk at the entry point of my OS and it differs from the expected checksum.

[Compaszer]

"Welcome to the classic OSDev rite of passage: 'It works in QEMU, but breaks on real hardware.'

1. Grub2 is not allowed to place modules overlapping those holes ore reserved regions as far as I know, so I don't think that this is the problem.
2. I have included this tag in the mulitboot2 header and the initramdisk is indeed page aligned.
3. I have used linker symbols to verify that the kernel size of the loaded kernel meets my expectations and that the kernel end has a distance of around 800KiB to the module.

[ZeSystem]

Is your system good right now?

[Compaszer]

What do you mean by this? Currently the initramdisk is corrupted. When I remove the include_bytes! macros, the initramdisk is ok and the system is running correctly.

[ZeSystem]

What do you mean by this? Currently the initramdisk is corrupted. When I remove the include_bytes! macros, the initramdisk is ok and the system is running correctly.

A380-sized misunderstanding, my bad! I meant to ask if the system runs perfectly only when the macro is gone, or if you managed to find a workaround.Since GRUB behaves and your bounds check out (800KiB gap is plenty), the issue might be real hardware limitations with GRUB2's memory allocation on that specific machine. Some older or buggy UEFI/BIOS firmware implementations fail to parse the memory map correctly when the ELF sections grow past certain boundaries, leading to GRUB overlapping the initrd with something else (like ACPI or firmware runtime code) despite what the specification says.Have you tried checking if the corrupted bytes in your checksum match any specific pattern? (e.g., all zeros, or filled with firmware/BIOS text strings?)

[Compaszer]

I have not checked that but I will try that.
But the board is an old enterprise server board, I somewhat doubt that it's really the firmware or grub that causes the issue.
But I will check.

[Compaszer]

I've just moved my kernel from 1MiB to 25MiB, resulting in the multiboot module getting located close above 1MiB. Now I know that the range where the module now resides isn't occupied by something else, as my kernel previously was at the same location. Still, the module appears to be corrupted. Interestingly the errors seem to be starting around 3308600B into the initramdisk independent of wether my kernel is at 1MiB or 25MiB.
Some of the corrupted data contain contiguous chunks of zeros and the initramdisk has the same (corrupted) bytes independent of the location it is placed at, which is very weird. I've double checked the initramdisk in the iso file and it is definetly correct there.

[Octocontrabass]

What happens if you load more modules? Does the corruption occur in the same part of the initramdisk if GRUB loads another module before it?

It sounds an awful lot like you've got a memory allocation bug of some sort. Are you sure your Rust entry point doesn't implicitly rely on your heap allocator?