OSDev.org

The Place to Start for Operating System Developers
It is currently Mon Jul 23, 2018 9:40 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 52 posts ]  Go to page Previous  1, 2, 3, 4  Next
Author Message
 Post subject: Re: Higher Half in C?
PostPosted: Thu Jul 12, 2018 12:20 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
simeonz wrote:
Octacone wrote:
It didn't fix the problem (aligning). BSS is still getting marked as PROGBITS...
The thing is - I cannot reproduce this on my end, so it is difficult to come up with a solution. But if you have found the crash issue and are willing to live with some extra zeroes.. fine :)


That is very strange. I would like to fix it doe, loading zeros is not useful by any means.
Ignoring all this, why the F does it take 10 F minutes for Bochs to load my kernel. This makes any kind of debugging almost impossible.
Imagine if you had to wait for Windows to load and it took that much.
Btw Bochs now (after fixing my ELF loader) breaks on insd dword ptr es:[edi], dx and ES is set to 0xFF, I think my kernel might be possessed.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Thu Jul 12, 2018 3:15 pm 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
simeonz wrote:
Octacone wrote:
It didn't fix the problem (aligning). BSS is still getting marked as PROGBITS...
The thing is - I cannot reproduce this on my end, so it is difficult to come up with a solution. But if you have found the crash issue and are willing to live with some extra zeroes.. fine :)


That is very strange. I would like to fix it doe, loading zeros is not useful by any means.
Ignoring all this, why the F does it take 10 F minutes for Bochs to load my kernel. This makes any kind of debugging almost impossible.
Imagine if you had to wait for Windows to load and it took that much.
Btw Bochs now (after fixing my ELF loader) breaks on insd dword ptr es:[edi], dx and ES is set to 0xFF, I think my kernel might be possessed.
If I were you, I would try to trigger breakpoint in Bochs programmatically and continue debugging from there. Use "asm volatile("xchg %bx, %bx" ::: "memory");" in the C code and "xchg %bx, %bx" in the assembly code. Check out the wiki page for the magic breakpoint configuration. Check out Bochs documentation and more specifically the symbols support (which is more like a custom map file that you must generate with nm).


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Fri Jul 13, 2018 3:09 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
simeonz wrote:
If I were you, I would try to trigger breakpoint in Bochs programmatically and continue debugging from there. Use "asm volatile("xchg %bx, %bx" ::: "memory");" in the C code and "xchg %bx, %bx" in the assembly code. Check out the wiki page for the magic breakpoint configuration. Check out Bochs documentation and more specifically the symbols support (which is more like a custom map file that you must generate with nm).


I've been using that functionality since the day one, it is super neat.
What I have figured so far is that once the jump happens there is some random data at that location that gets executed, probably my fault. I need to investigate that further.
Kernel not getting copied in the right spot... This is exhausting.
I really don't want to copy 800000 zeros for nothing.
Any more ideas on that?

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 1:42 am 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
I really don't want to copy 800000 zeros for nothing.
Any more ideas on that?
First - I would check the input object files with readelf -S, and verify that their bss sections are all NOBITS. If they are not, the investigation has to focus on that odd one from the bunch. If all of them are NOBITS, split the higher bss section in the linker script in two - one for the bss data and one for the common symbols. Align them both on a page boundary just in case and try switching their positions as well. The idea here is to check if the bss section or the common symbols are the cause for the PROGBITS on their own.

P.S. I realize that this is a lot of toiling, but I do not recognize the problem. I can only offer investigation hints. If someone else recognizes the issue, hopefully they will speak up.


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 1:48 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
simeonz wrote:
Octacone wrote:
I really don't want to copy 800000 zeros for nothing.
Any more ideas on that?
First - I would check the input object files with readelf -S, and verify that their bss sections are all NOBITS. If they are not, the investigation has to focus on that odd one from the bunch. If all of them are NOBITS, split the higher bss section in the linker script in two - one for the bss data and one for the common symbols. Align them both on a page boundary just in case and try switching their positions as well. The idea here is to check if the bss section or the common symbols are the cause for the PROGBITS on their own.


They are shown as PROGBITS. I will try what you suggested and report back.
Edit: btw lower half bss is all good and marked as NOBITS. That was never a problem.
Edit 2: nothing worked until I excluded *(.bss) from .bss_higher_half and then I checked again and it indeed was marked as NOBITS but was placed after .bss_lower_half which is not good. I'll try to make it a standalone section.
Edit 3: fixed, I'm something of a scientist myself. :D I just did this: section .bss_higher_half nobits inside my assembly file and boom, marked as NOBITS! That was an easy fix. Looks like the linker only marks something as NOBITS if it's called .bss or explicitly declared as such lol.
Now I need to figure out why the higher half won't work.
Edit 4: loading times improved by 100000x.
Edit 5: I found out my problem:
Code:
for(uint32_t current_address = 0; current_address <= 0x1000000; current_address += 0x1000)
{
       Map_Virtual_To_Physical(current_address, current_address, true, true, false);
       Map_Virtual_To_Physical(0xC0000000 + current_address, current_address, true, true, false);
}

That is not going to work.
I checked with Bochs and page 0xC0000000 maps to 0x00000000, which means that the CPU will actually jump at 0x0 in reality an execute garbage. Now I need to figure out how to solve this. There must be an offset that I have to add to 0xC0000000 + current_address to make it work.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 4:08 am 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
They are shown as PROGBITS. I will try what you suggested and report back.
I assume you mean that .bss_higher_half in Kernel_Assembly.o was PROGBITS? From your following discoveries, I gather that the .bss sections of all input files were NOBITS (except in the output).
Octacone wrote:
Edit 3: fixed, I'm something of a scientist myself. :D I just did this: section .bss_higher_half nobits inside my assembly file and boom, marked as NOBITS! That was an easy fix. Looks like the linker only marks something as NOBITS if it's called .bss or explicitly declared as such lol.
Actually, originally you had only .bss in the assembly. In any case, this is documented behavior. The assembler guesses the "type" of the section if has a "standard" name, but otherwise you need to specify the type (such as nobits) and the access flags explicitly if you need to.
Octacone wrote:
Edit 5: I found out my problem:
Code:
for(uint32_t current_address = 0; current_address <= 0x1000000; current_address += 0x1000)
{
       Map_Virtual_To_Physical(current_address, current_address, true, true, false);
       Map_Virtual_To_Physical(0xC0000000 + current_address, current_address, true, true, false);
}


That is not going to work.
I checked with Bochs and page 0xC0000000 maps to 0x00000000, which means that the CPU will actually jump at 0x0 in reality an execute garbage. Now I need to figure out how to solve this. There must be an offset that I have to add to 0xC0000000 + current_address to make it work.

Well. You have two options here. One is to identity map the space between 1M and the end of the kernel and to map the addresses between higher_physical_start and the end of the kernel to the higher half, like this:
Code:
for(uint32_t physical_address = 0x100000; physical_address < higher_physical_start; physical_address += 0x1000)
{
       Map_Virtual_To_Physical(physical_address , physical_address , true, true, false);
}
for(uint32_t virtual_address = 0xC0000000; virtual_address < kernel_end; virtual_address += 0x1000)
{
       /* This can be optimized, but the compiler might optimize it automatically. */
       uint32_t physical_address = virtual_address - 0xC0000000 + higher_physical_start;
       Map_Virtual_To_Physical(physical_address, physical_address, true, true, false);
       Map_Virtual_To_Physical(virtual_address, physical_address, true, true, false);
}
This means that you have to subtract 0xC0000000 and add higher_physical_start if you ever want to compute the physical address of a page in the higher kernel. (You may not need this.)

Alternatively, you could have the identity mapping as before, but map higher_physical_start to the end of the kernel to the higher half at the same offset as the physical address. You need to modify the linker script:
Code:
ENTRY(Pre_Kernel_Main_Lower_Half)
OUTPUT_FORMAT("elf32-i386")
SECTIONS
{
    . = 0x100000;
    .text_lower_half :
    {
        Objects/Kernel_Assembly.o(.text_lower_half)
        Objects/Higher_Half.o(.text)
    }
    .rodata_lower_half :
    {
        Objects/Higher_Half.o(.rodata)
    }
    .data_lower_half :
    {
        Objects/Kernel_Assembly.o(.data_lower_half)
        Objects/Higher_Half.o(.data)
    }
    .bss_lower_half :
    {
        bss_start_lower_half = .;
        Objects/Higher_Half.o(.bss)
        Objects/Higher_Half.o(COMMON)
        bss_end_lower_half = .;
    }
    higher_physical_start = ABSOLUTE(ALIGN(0x1000);
    . = 0xC0000000 + higher_physical_start;
    kernel_start = ABSOLUTE(.);
    .text_higher_half : AT(higher_physical_start)
    {
        Objects/Kernel_Assembly.o(.text_higher_half)
        *(.text)
    }
    .rodata_higher_half :
    {
        start_constructors = .;
        *(SORT(.ctors*))
        end_constructors = .;
        *(.rodata)
    }
    .data_higher_half :
    {
        *(.data)
    }
    .bss_higher_half :
    {
        bss_start_higher_half = .;
        Objects/Kernel_Assembly.o(.bss_higher_half)
        *(.bss)
        *(COMMON)
        bss_end_higher_half = .;
    }
    kernel_end = ABSOLUTE(.);
}
And the mapping code would look like this:
Code:
for(uint32_t physical_address = 0x100000; physical_address < higher_physical_start; physical_address += 0x1000)
{
       Map_Virtual_To_Physical(physical_address , physical_address , true, true, false);
}
for(uint32_t virtual_address = kernel_start; virtual_address < kernel_end; virtual_address += 0x1000)
{
       /* This can be optimized, but the compiler might optimize it automatically. */
       uint32_t physical_address = virtual_address - 0xC0000000;
       Map_Virtual_To_Physical(physical_address, physical_address, true, true, false);
       Map_Virtual_To_Physical(virtual_address, physical_address, true, true, false);
}


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 4:53 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
@simeonz
Man, many many thanks for helping me out, but I think I'll just have to give up.
Looks like this battle was lost long before it was even started.
I'm getting division by zero for no reason (only when I don't zero out lower half BSS, when I do it triple faults). I've been working on this for a week and nothing has changed.
Looks like my zero lower half BSS code overwrites first 2 bytes at 0x906000(physical address of higher half).
I guess I will just have to ditch the entire higher half idea.
Thanks to all the people that tried to help too.
Just wasting my (yours too) time.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 5:21 am 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
@simeonz
Man, many many thanks for helping me out, but I think I'll just have to give up.
Looks like this battle was lost long before it was even started.
I'm getting division by zero for no reason. I've been working on this for a week and nothing has changed.
I guess I will just have to ditch the entire higher half idea.
Thanks to all the people that tried to help too.
Just wasting my (yours too) time.
First, I just realized that you will need to switch your IDT, GDT, TSS before unmapping the lower half.

Second, it isn't like you were stumbling on a hello world assignment in javascript. Take a break, stuff the project in a "drawer", etc. It is your choice. Division by zero doesn't sound that terrible. But in any case, even if you decide to quit, don't sweat it.


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 5:34 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
simeonz wrote:
Octacone wrote:
@simeonz
Man, many many thanks for helping me out, but I think I'll just have to give up.
Looks like this battle was lost long before it was even started.
I'm getting division by zero for no reason. I've been working on this for a week and nothing has changed.
I guess I will just have to ditch the entire higher half idea.
Thanks to all the people that tried to help too.
Just wasting my (yours too) time.
First, I just realized that you will need to switch your IDT, GDT, TSS before unmapping the lower half.

Second, it isn't like you were stumbling on a hello world assignment in javascript. Take a break, stuff the project in a "drawer", etc. It is your choice. Division by zero doesn't sound that terrible. But in any case, even if you decide to quit, don't sweat it.


Don't worry about the GDT, IDT, TSS and stuff because that would all be initialized within Kernel_Main before enabling paging (for the second time). Also there was no unmapping of the lower half at all since the kernel would never reach that point.
Well I don't know. I'm starting to realize that doing this in C++ causes more problems than it solves. Maybe I should really do it in assembly.
I just don't get how people can do that in like 10 lines of code. You would have to create a page directory, page table entry, page tables, pages link all that goo together and then map everything and stuff, just how can people do that in like 5 lines of code by moving some magic values around. That is the problem I'm having with assembly paging what do those magic values do? Like 0x00000083 and such... I mean sure I could just copy paste something but that is not how you learn and that is not my goal. Memory management is important you really want to understand every part of it.
Also is switching from 4 MB to 4 KB pages actually allowed?

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 7:25 am 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
I'm starting to realize that doing this in C++ causes more problems than it solves. Maybe I should really do it in assembly.
If it tickles your fancy. :) I doubt that the language matters.
Octacone wrote:
I just don't get how people can do that in like 10 lines of code. You would have to create a page directory, page table entry, page tables, pages link all that goo together and then map everything and stuff, just how can people do that in like 5 lines of code by moving some magic values around. That is the problem I'm having with assembly paging what do those magic values do? Like 0x00000083 and such... I mean sure I could just copy paste something but that is not how you learn and that is not my goal. Memory management is important you really want to understand every part of it.
They might be working in protected mode without paging. Or they might be using a single large page to map the beginning of the address space. But there are caveats regarding the use of INVLPG against the first 4M/1M. (There is some info in another post on this forum.) And literal constants are much less readable when compared to symbolic constants. (Regarding 0x00000083 - I can't help you there. Where is this used - CR4, the pte?)
Octacone wrote:
Also is switching from 4 MB to 4 KB pages actually allowed?
You can mix 4K and 4M in the address space, obviously. And you can switch a virtual address between the two, assuming that you invalidate the TLB first (with INVLPG).

Here is a small portion of the initial linux boot code. See how sleek and small it is. And calls quite a few C code pieces as well.


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 8:22 am 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
simeonz wrote:
Octacone wrote:
I'm starting to realize that doing this in C++ causes more problems than it solves. Maybe I should really do it in assembly.
If it tickles your fancy. :) I doubt that the language matters.
Octacone wrote:
I just don't get how people can do that in like 10 lines of code. You would have to create a page directory, page table entry, page tables, pages link all that goo together and then map everything and stuff, just how can people do that in like 5 lines of code by moving some magic values around. That is the problem I'm having with assembly paging what do those magic values do? Like 0x00000083 and such... I mean sure I could just copy paste something but that is not how you learn and that is not my goal. Memory management is important you really want to understand every part of it.
They might be working in protected mode without paging. Or they might be using a single large page to map the beginning of the address space. But there are caveats regarding the use of INVLPG against the first 4M/1M. (There is some info in another post on this forum.) And literal constants are much less readable when compared to symbolic constants. (Regarding 0x00000083 - I can't help you there. Where is this used - CR4, the pte?)
Octacone wrote:
Also is switching from 4 MB to 4 KB pages actually allowed?
You can mix 4K and 4M in the address space, obviously. And you can switch a virtual address between the two, assuming that you invalidate the TLB first (with INVLPG).

Here is a small portion of the initial linux boot code. See how sleek and small it is. And calls quite a few C code pieces as well.


Okay so I deleted all the C higher half code and officially decided to do it in Assembly.
I figure everything out except what does a 4 MB page look like, is it the same as 4 KB page. Cannot find that anywhere, Intel manuals only mention page directory entry that maps a 4 MB page.
I need to identity map everything from 0 to 12 MB and also higher half map that same range, which means I need 6 (4 MB) pages.
Just to be sure, this is how 4 MB paging works:
CR3 -> Page Directory (aligned physical address) -> contains 1024 page directory entries that are each 4 bytes big -> each of those entries points to a page that is 4 bytes big
I know I need to enable PSE.
I also know that the pages I need are: 0, 1, 2, 768, 769, 770 -> can you confirm this? (0-3 MB, 4-8 MB, 8-12 MB) (3GB.0MB - 3GB.3MB, 3GB.4MB - 3GB.8MB, 3GB.8MB - 3GB.12MB)
When I do all that Bochs doesn't see any page tables, paging is enable, cr3 and cr0 and cr4 are loaded but no mappings. Can you just confirm that my ideas are OK. No I didn't forget to subtract 0xC0000000 from all the addresses.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sat Jul 14, 2018 10:12 am 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
CR3 -> Page Directory (aligned physical address) -> contains 1024 page directory entries that are each 4 bytes big -> each of those entries points to a page that is 4 bytes big
I know I need to enable PSE.
I also know that the pages I need are: 0, 1, 2, 768, 769, 770 -> can you confirm this? (0-3 MB, 4-8 MB, 8-12 MB) (3GB.0MB - 3GB.3MB, 3GB.4MB - 3GB.8MB, 3GB.8MB - 3GB.12MB)
When I do all that Bochs doesn't see any page tables, paging is enable, cr3 and cr0 and cr4 are loaded but no mappings. Can you just confirm that my ideas are OK. No I didn't forget to subtract 0xC0000000 from all the addresses.
Assuming that you are not using PAE for the time being. Yes. The above calculations are correct. 4MB pages are pointed to directly by the page directory. 4KB pages are pointed to by the page tables. To indicate a 4MB page, you have to set the PS bit (bit 7) in the page directory entry. Otherwise, the entry is considered a pointer to a page table. So, you need to fill entries 0, 1, 2, 768, 769, 770 of the page directory and make sure that their PS bit is set. (In the Intel manual there is a "Format of a 32-Bit Page-Directory Entry that Maps a 4-MByte Page" table.)


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sun Jul 15, 2018 12:08 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
Success!!!
I managed to enable PSE 4 MB paging in assembly and identity map 1 MB to 1 MB and 3 GB to 1 MB. Everything seems to be working fine. My kernel is officially in higher half.
Now I'm having a problem because it crashes when I try to re-enable paging, more specially turning off PSE and turning on PAE and resetting CR3. How to solve that?

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sun Jul 15, 2018 3:03 pm 
Offline
Member
Member

Joined: Fri Aug 19, 2016 10:28 pm
Posts: 348
Octacone wrote:
Success!!!
I managed to enable PSE 4 MB paging in assembly and identity map 1 MB to 1 MB and 3 GB to 1 MB. Everything seems to be working fine. My kernel is officially in higher half.
Now I'm having a problem because it crashes when I try to re-enable paging, more specially turning off PSE and turning on PAE and resetting CR3. How to solve that?
When you turn off PSE, the current code page (being large 4MB page) will be immediately discarded from the TLB, then resolved again, but since PSE is off, the PS bit will be ignored, and the PDE will be interpreted as a pointer to a page table. This is sufficient for a crash. If you turn on the PAE, the paging structures immediately switch to 3-level with PDPTEs at the top. I don't think that there is a way to do that safely without disabling paging first, in which case you are back to square one. I would advise you to keep PSE. Or don't map the kernel with large pages. If you want PAE, you better start with PAE on. There will be one more level of indirection, and you will have to set the first directory entry of the first and fourth page directories.


Top
 Profile  
 
 Post subject: Re: Higher Half in C?
PostPosted: Sun Jul 15, 2018 3:14 pm 
Offline
Member
Member
User avatar

Joined: Fri Aug 07, 2015 6:13 am
Posts: 957
simeonz wrote:
Octacone wrote:
Success!!!
I managed to enable PSE 4 MB paging in assembly and identity map 1 MB to 1 MB and 3 GB to 1 MB. Everything seems to be working fine. My kernel is officially in higher half.
Now I'm having a problem because it crashes when I try to re-enable paging, more specially turning off PSE and turning on PAE and resetting CR3. How to solve that?
When you turn off PSE, the current code page (being large 4MB page) will be immediately discarded from the TLB, then resolved again, but since PSE is off, the PS bit will be ignored, and the PDE will be interpreted as a pointer to a page table. This is sufficient for a crash. If you turn on the PAE, the paging structures immediately switch to 3-level with PDPTEs at the top. I don't think that there is a way to do that safely without disabling paging first, in which case you are back to square one. I would advise you to keep PSE. Or don't map the kernel with large pages. If you want PAE, you better start with PAE on. There will be one more level of indirection, and you will have to set the first directory entry of the first and fourth page directories.


I definitely don't want to keep PSE. This never gets easy, doesn't it?
This is going to get really messy and complicated. Making PAE work in assembly, great, just great. Shouldn't take more than like 5 weeks, but OK.

_________________
OS: Basic OS
About: 32 Bit Monolithic Kernel Written in C++ and Assembly, Custom FAT 32 Bootloader


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 52 posts ]  Go to page Previous  1, 2, 3, 4  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: Schol-R-LEA and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group