OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Oct 22, 2019 1:41 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 12 posts ] 
Author Message
 Post subject: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 9:30 am 
Offline

Joined: Thu Apr 28, 2016 9:18 am
Posts: 3
I've been struggling for this for over a week and a half...

I'm trying to parse the multiboot structure passed by the bootloader to my kernel in order to implement memory allocation for paging. I've followed the example from the multiboot specifications (https://www.gnu.org/software/grub/manua ... rnel_002ec), but the memory map from GRUB is invalid... all I'm getting is type = 0. I've tried booting from both bin (qemu-system-i386 -kernel myos.bin) and iso (qemu-system-i386 -cdrom myos.iso) but none of them give me a correct memory map.

Here is the output

Image

Here is the assembly that calls kernel_main:

Code:
_start:
    ; To set up a stack, we simply set the esp register to point to the top of
    ; our stack (as it grows downwards).
    mov esp, stack_top

    ; Push pointer to the Multiboot information structure
    push ebx

    extern kernel_main
    call kernel_main

    cli
.hang:
    hlt
    jmp .hang



Here is how I'm parsing the multiboot header:

Code:
    if (mbi->flags & MULTIBOOT_INFO_MEMORY)
    {
        vga_printf("mem_lower = %uKB, mem_upper = %uKB\n",
                (uint32_t)mbi->mem_lower, (uint32_t)mbi->mem_upper);
    }

    if (mbi->flags & MULTIBOOT_INFO_MEM_MAP)
    {
        vga_printf("mmap_addr = 0x%x, mmap_length = 0x%x\n",
                (uint32_t)mbi->mmap_addr, (uint32_t)mbi->mmap_length);

        for (mmap = (struct multiboot_mmap_entry*)mbi->mmap_addr;
                (uint32_t)mmap < (mbi->mmap_addr + mbi->mmap_length);
                mmap = (struct multiboot_mmap_entry*)((uint32_t)mmap
                    + mmap->size + sizeof(mmap->size)))
        {
            vga_printf("base_addr_high = 0x%x, base_addr_low = 0x%x, "
                    "length_high = 0x%x, length_low = 0x%x, type = 0x%x\n",
                    mmap->addr >> 32,
                    mmap->addr & 0xFFFFFFFF,
                    mmap->len >> 32,
                    mmap->len & 0xFFFFFFFF,
                    (uint32_t)mmap->type);
        }
    }


Anybody have any ideas? I've looked everywhere, but I can't seem to find a solution for this (as I said, I've been struggling on this for a week and a half).


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 9:40 am 
Offline
Member
Member

Joined: Sat Nov 10, 2012 1:16 pm
Posts: 62
I suspect your multiboot_mmap_entry struct is aligning uint64_t to 8-byte multiples, which means fields are all getting put in the wrong spots.

Evidence for this is that sane values for length_low are showing up in length_high instead.

You'll want to pack the struct.


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 10:30 am 
Offline

Joined: Thu Apr 28, 2016 9:18 am
Posts: 3
MDenham wrote:
I suspect your multiboot_mmap_entry struct is aligning uint64_t to 8-byte multiples, which means fields are all getting put in the wrong spots.

Evidence for this is that sane values for length_low are showing up in length_high instead.

You'll want to pack the struct.


Hey, thanks for the response! I took the multiboot header from the specifications... here is the part where they defined the multiboot_mmap_entry structure:

Code:
struct multiboot_mmap_entry
{
    multiboot_uint32_t size;
    multiboot_uint64_t addr;
    multiboot_uint64_t len;
#define MULTIBOOT_MEMORY_AVAILABLE              1
#define MULTIBOOT_MEMORY_RESERVED               2
    multiboot_uint32_t type;
} __attribute__((packed));
typedef struct multiboot_mmap_entry multiboot_memory_map_t;


It seems like they did the structure aligning like you suggested. Is there anything that I'm missing?


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 11:11 am 
Offline
Member
Member

Joined: Sat Nov 10, 2012 1:16 pm
Posts: 62
This kind of thing came up in another thread recently, where the compiler didn't appear to be obeying __attribute__((packed)) correctly with respect to uint64_t (and __attribute__((aligned (4))) didn't seem to fix the issue either).

The only suggestion I have is making a copy of the structure that uses 2x uint32_t in place of one uint64_t and then dropping the bitshift/mask stuff. You'll need to put values back together if you're planning on long mode support, but that's easy enough.


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 1:09 pm 
Offline

Joined: Thu Apr 28, 2016 9:18 am
Posts: 3
MDenham wrote:
This kind of thing came up in another thread recently, where the compiler didn't appear to be obeying __attribute__((packed)) correctly with respect to uint64_t (and __attribute__((aligned (4))) didn't seem to fix the issue either).

The only suggestion I have is making a copy of the structure that uses 2x uint32_t in place of one uint64_t and then dropping the bitshift/mask stuff. You'll need to put values back together if you're planning on long mode support, but that's easy enough.


Beautiful. This did the trick! Thanks so much... I can't believe how long I struggled on this for. I can finally get back to developing paging for my kernel. Do you have any insight on why the compiler incorrectly packs the unsigned long long?

Here is the memory map parsed correctly:

Image


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 1:42 pm 
Offline
Member
Member

Joined: Sat Nov 10, 2012 1:16 pm
Posts: 62
I don't know why it's doing it, to be honest. The closest thing I've found to anything relevant is something involving alignof() and that was declared "not a bug".

I suspect the reason for it might have to do with C++11/14 support, but I'm just spitballing at that point.

EDIT: Not sure what exactly the gcc switches are to get it to give you "here's what your C++ code looks like in assembly", but I can at least confirm that, despite all its other shortcomings, Visual Studio 2015 does not cause this problem to happen by looking at the assembly it produces for essentially that code (different variable names is the main difference). (It does, however, produce ridiculously bloated assembly. Instructions that are effective no-ops like "shl ecx, 0" show up, for example.)


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 2:35 pm 
Offline
Member
Member

Joined: Fri May 01, 2015 2:23 am
Posts: 63
Are you using MinGW by any change? "IA-32/x86-64 Windows mingw targets are using the -mms-bitfields option by default." Which means packing obeys m$ rules. To force it otherwise, use __attribute__((packed,gcc_struct)) or -mno-ms-bitfields at command line.

_________________
Hellbender OS at github.


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 2:50 pm 
Offline
Member
Member

Joined: Sat Nov 10, 2012 1:16 pm
Posts: 62
Hellbender wrote:
Are you using MinGW by any change? "IA-32/x86-64 Windows mingw targets are using the -mms-bitfields option by default." Which means packing obeys m$ rules. To force it otherwise, use __attribute__((packed,gcc_struct)) or -mno-ms-bitfields at command line.
See my edit above: if it were using MS packing rules, it wouldn't be forcing the uint64_t onto an 8-byte boundary despite __attribute__((packed)) (which is what the problem is), because VS2015 code doesn't force that unless requested.

Hell, I don't even think my structure is packed and VS2015's still letting uint64_t's be unaligned. I should go make sure it's marked as packed anyway, just in case.

(That said, I'm pretty sure this is gcc's doing, just because 99% of the people writing code here are compiling with gcc.)


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 5:19 pm 
Offline
Member
Member

Joined: Tue Feb 10, 2015 3:36 pm
Posts: 47
MDenham wrote:
I don't know why it's doing it, to be honest. The closest thing I've found to anything relevant is something involving alignof() and that was declared "not a bug".

I suspect the reason for it might have to do with C++11/14 support, but I'm just spitballing at that point.

EDIT: Not sure what exactly the gcc switches are to get it to give you "here's what your C++ code looks like in assembly", but I can at least confirm that, despite all its other shortcomings, Visual Studio 2015 does not cause this problem to happen by looking at the assembly it produces for essentially that code (different variable names is the main difference). (It does, however, produce ridiculously bloated assembly. Instructions that are effective no-ops like "shl ecx, 0" show up, for example.)

Careful there, it depends on how you set the optimizations options in the project properties. And some nops may appear because the compiler tries to align the code in some way. It saves you from a lot of trouble to use #pragma pack() on structures that must respect a certain layout.
Note that on some settings the compiler may generate SSE instructions (if you build for x64 it considers that SSE is a given thing) and that might crush your kernel in the early stages, also take care on that.


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Thu Apr 28, 2016 5:47 pm 
Offline
Member
Member

Joined: Sat Nov 10, 2012 1:16 pm
Posts: 62
zdz wrote:
MDenham wrote:
I don't know why it's doing it, to be honest. The closest thing I've found to anything relevant is something involving alignof() and that was declared "not a bug".

I suspect the reason for it might have to do with C++11/14 support, but I'm just spitballing at that point.

EDIT: Not sure what exactly the gcc switches are to get it to give you "here's what your C++ code looks like in assembly", but I can at least confirm that, despite all its other shortcomings, Visual Studio 2015 does not cause this problem to happen by looking at the assembly it produces for essentially that code (different variable names is the main difference). (It does, however, produce ridiculously bloated assembly. Instructions that are effective no-ops like "shl ecx, 0" show up, for example.)

Careful there, it depends on how you set the optimizations options in the project properties. And some nops may appear because the compiler tries to align the code in some way. It saves you from a lot of trouble to use #pragma pack() on structures that must respect a certain layout.
Note that on some settings the compiler may generate SSE instructions (if you build for x64 it considers that SSE is a given thing) and that might crush your kernel in the early stages, also take care on that.
Yeah, right now it's set with no optimizations at all. Didn't think about it trying to align the code, though. That'd explain doubled jumps as well.

I'm still planning on going through the assembly it says it's producing and hand-tuning that, but that's more out of OCD than anything else. :-)


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Fri Apr 29, 2016 2:25 am 
Offline
Member
Member

Joined: Sun Feb 01, 2009 6:11 am
Posts: 1067
Location: Germany
MDenham wrote:
I'm still planning on going through the assembly it says it's producing and hand-tuning that, but that's more out of OCD than anything else. :-)

When you hand-tune the compiler output, never forget to actually measure the results. Because if you don't, chances are that you're making things worse rather than better. (Well, you always make things worse in terms of maintainability, but you seem to care more about performance or code size, and there it's only almost always. Assuming that you do enable optimisations before you criticise the compiler output, of course.)

_________________
Developer of tyndur - community OS of Lowlevel (German)


Top
 Profile  
 
 Post subject: Re: QEMU Multiboot Invalid Memory Map
PostPosted: Fri Apr 29, 2016 2:38 am 
Offline
Member
Member

Joined: Tue Feb 10, 2015 3:36 pm
Posts: 47
Kevin wrote:
MDenham wrote:
I'm still planning on going through the assembly it says it's producing and hand-tuning that, but that's more out of OCD than anything else. :-)

When you hand-tune the compiler output, never forget to actually measure the results. Because if you don't, chances are that you're making things worse rather than better. (Well, you always make things worse in terms of maintainability, but you seem to care more about performance or code size, and there it's only almost always. Assuming that you do enable optimisations before you criticise the compiler output, of course.)

Except for some corner cases I don't think one can do better than the compiler. And even then, the gain in speed / size is almost irrelevant.
Apart from curiosity and to rule out some bugs (or to make sure that a specific sequence of instructions is in the order I want it do be) I never look at the generated assembly (it's interesting to compare it for different optimization settings when you first start to play with that). If I need something done in assembly by me I do it in assembly (it's usually profiling code, or some weird stuff).


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: MSN [Bot] and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group