OSDev.org

Hi,



You shouldn't blame the original designer's of "E820" for this. When PCI was first created CPUs (80486) only supported 32-bit physical addresses, and therefore the original PCI devices had to use 32-bit physical addresses. Later (when things like PAE were introduced) it was too late - PCI devices were already using 32-bit physical addresses, so there had to be a "PCI hole".

The only other alternative would have been to shift all memory mapped PCI devices above 4 GiB. In this case a "32-bit physical addresses only" OS would still waste (e.g.) 28 GiB of RAM when there's 32 GiB of RAM installed, but (as an added bonus) it also wouldn't be able to use any memory mapped PCI device.



Why? Typically you just need to add a few "if(PAE) { ... } else { ... }" and a little extra code in a few places (in whatever function/s you use to map and unmap pages, and in the page fault handler).



It should take less than 1 week to implement and debug.



It's ancient stuff from over 20 years ago. Every emulator has supported it for decades.



Some CPUs (e.g. cheap Atom CPUs intended for small embedded systems) only support 32-bit physical addresses. With PAE you'd be able to access 4 GiB or more physical space (up to 4096 TiB), depending on what the CPU supports. Of course the physical address space contains more than just RAM, so the amount of RAM installed is always less than the amount of space the CPU supports.

Also note that PAE supports "no execute" page protection (while "plain 32-bit paging" doesn't); which means that you get better security too. For disadvantages, page table entries (and page directory entries, etc) are twice as large, so PAE costs a little extra memory.



This is part of why "map all RAM into kernel space" is something I consider a beginner's mistake.



Small pages are still 4 KiB. Large pages become 2 MiB (rather than 4 MiB).



For someone like me (who has experience using PAE, separates "physical memory management" from "virtual memory management", etc) it'd be about 1 day. For you, I'd add an extra 2 days learning about it (especially if you're using the "recursive mapping trick", which gets a little messy for PAE), then add a few more days for debugging (the first time there's always a bug that takes a while to figure out); and then I'd round it up to 1 week (because estimates of how long any software will take to be "done" are always wrong).

I'd also say that it's one of those things where it's easier to do it sooner (before a whole bunch of other code is added on top) and harder to do it later; which means that it's better to do it now (regardless of how long it takes to do).


Cheers,

Brendan

What Brendan said and I just wanted to add that under 32-bit only a single process should be restricted to 4GiB of VAS and thus vmem, and that vmem can be backed by pmem (RAM), and thus if you have ten processes each using 4GiB you can utilize 40GiB of pmem.

If you want to get technical (I probably wouldn't suggest it though), you can do "banking", where the app signals the OS to switch for example 1GiB of the VAS to a different pmem, that way a single process can use more than 4GiB of RAM, but it gets tricky and you have to be aware how your compiler works, and I'd consider the benefits marginal, especially for a hobby OS. I'm not sure if any OS has actually done that. Simply put, if a process needs more than 4GiB of memory, then it should be running 64-bit.

Note, you might not like this, but if you have 64-bit hardware this might also be a good point to consider long mode. The reality is protected mode is legacy and for most purposes obsolete these days. Just wanted to point it out before you get even more invested in prot-mode, and a month or two from now you'll curse for something else =)

Not exactly banking, in the sense that it is not intended for swapping parts of the code segment in and out of view, but for data there is something similar in Windows. The way I see it, its use requires a virtual region manager implemented inside the application, which is a little clumsy. But still, it is part of a mainstream OS.

For the PAE discussion, I will repeat myself (from another recent post), but Windows utilizes the extra memory for file cache. The filesystem blocks can occupy physical memory without being mapped. The OS maps the data into view when necessary to complete IO. The result is virtual memory thrashing, without physical memory thrashing. (But the maps persist until they are evicted on a pseudo-lru basis, just like physical memory management, but with larger than page granularity.)

@Branden, @LtG @simeonz

Thanks for replying. I roughly know what to do. I don't want to get all technical, yet. There is not need for anything advanced because there is no multitasking, processes, execution stuff, etc... implemented.
I definitely do not want to consider long mode. I am not ready for it. Paging would get much worse. I would virtually have no debugging, can't print whole 64 bit integers, etc... GRUB requires hacks in order to load a 64 bit kernel, not messing with that and the most important thing Virtual 8086 mode is not supported (the most important thing ever created, ever).
I'll be sticking with 32 bit protected mode for a while(true) and I'll try to enable and implement PAE.
Edit: I don't quite understand 2 MB stuff, I only and exclusively want to use 4 KB pages.

I love how often and in different ways people manage to misspell Brendan =)

V86 is essentially completely useless these days, and while can't be directly used from long mode you can drop back into prot-mode for V86, but since it's useless you never want to.

Grub hacks aren't needed if you take the handoff from grub in prot-mode and enable long mode yourself. Not sure what you mean by "much worse paging", I mean you're trying to enable PAE anyway, right? And I thought for the mmap you already have 64-bit printing? If not, you should fix that anyway.

Note, I'm not trying to push you into long mode, just trying to say that the difference isn't really that big and you'd get all the benefits. But if you prefer prot-mode, that's fine too.

You can continue using 4KiB pages, no need to use 2MiB pages. A page table used to hold 1024 entries, each entry being 4B. Because you want to address more than 32-bits of physical address space (64-bits) that means you can only fit 512 entries in one page table. Thus one page table can only address 2MiB of memory. If you use large pages then the page table doesn't exist and the upper level table/directory points directly to 4MiB/2MiB (32-bit/64-bit) of memory.

Or did you mean something else with the "2MB stuff"?

Just realized that is not how you write it! I always had to copy his name when replying. =)
Virtual 8086 is not useless, it is protected mode's proudest sub mode. Not available in long mode = currently not my problem because I'll stick with protected mode.
I have no idea on how that could work, protected mode legacy paging conversion into long mode paging while mode switching and higher half possibly. Very tangled up.
Worse paging means more/less paging structures, PML4 addition, meaning more changes and thus harder implementation.
I can only print 64 bit hexadecimal numbers by shifting bits around, can't do the same with integers doe.
Yes that is what I was talking about, that answers it very precisely. +1

AMD seems to think otherwise. Why do you want to use it so much, anyway?


No, but you could extend your function that prints 32-bit integers to handle 64-bit integers. (And if you say you can't, you're probably doing something wrong.)

Hello everybody!!!
For my OS, I have the output from the picture.
So, my question is: why there is no entry for the memory between 0x000A0000 and 0x000F0000? The memory address for text mode is 0xB8000 and is between the two addresses, so that memory area exists...

And another question (especially for Octacone): i'm in the same situation as you, i'm using an bitmap for physical memory manager and I would like to know how you calculated how much ram you have.

Why do you want to know how much RAM there is? So you can size your bitmap maybe?

Suppose there's an odd system out there that has 1GiB of memory at the very end of the 64-bit PAS (Physical Address Space) in addition to another 1GiB at the beginning, which means in the middle there's empty space. Are you going to create a bitmap the size of the entire 64-bit PAS? So basically if you want to use bitmaps then you should probably keep multiple bitmaps, each with info about the base address (start of the bitmap), so you know what address each bit in the bitmap refers to, instead of just one huge bitmap that covers empty space and always starts at PAS address 0.

Of course you can also just assume that no such systems exist, and waste space in your bitmap for "empty space"...

I think any memory regions not returned by GRUB can be considered a memory hole that isn't intended for regular use.

Hey @TheLittleWho
Basically what I've done is summed up all the lengths (note they are 64 bit wide) that are marked as free. Only count those free ones or else you'll register more RAM than physically available. Not all memory areas are stored in RAM. Some of them are memory mapped devices and other BIOS related mappings.
You have 2 (3 if you count long mode which I won't) options when it comes to your physical memory manager:

• Standard bitmap type, limited to 4 GB of physically addressable RAM (because 32 bits = 2^32 bits = 0xFFFFFFFF (0x100000000 - 1)). This does not mean you can use those 4 GB as you want. It is more likely to be 3.5 GB or 2 GB in total. Actively wasting an entire gigabyte, again memory mapped devices within 0xFFFFFFFF region = less usable RAM -> other half (0.5 GB or 1 GB) above 32 bit (4 GB) limit. Pros: simple to implement Cons: wasting huge amount of RAM.
• Side note: ACPI has some of its tables mapped withing 0xFFFFFFFF -> meaning you can see which those are (returned by GRUB/E820) -> after safely storing/using them you are free to mark those regions as free/usable memory.
• A slightly better option: the one I've chosen to implement. Enable PAE (physical address extension, something in those lines) and get access to the "upper upper" memory above 0xFFFFFFFF up to couple of TB, but that only reefers to physical memory access. You have to enable paging in order to use this method. You will still be limited to 4 GB of virtual address space but you will be able to map those 4 GB to any physical address you want. Then make your PMM super super smart. This is where it gets interesting... You can have a single bitmap (64 GB in size, assuming average maximum amount of physical memory that can be installed by user on a standard industry motherboard, server ones can have much more) which means you can just map your memory map accordingly and reserve all the areas that are not free. 64 GB = 64 * 1024 ^3 = 68719476736 / 4096 (4 KB pages) / 32 (32 bits in a bitmap (per uint32_t)) = 524288 bytes long bitmap = 512 KB / 0.5 MB. That is wasteful (@LtG thinks so.) What you could do is: have a set of different bitmaps each once for a different free region, allowing your master code to access those bitmaps as if they were contiguous, like check if there is space in bitmap no.1 then if there is no sufficient space (requested space) available search the next one (no.2) and so on. Virtually chaining those bitmap as if they were one. Also while doing that reclaim everything you can and store it somewhere safely, also make sure to map everything properly. If you get weird page faults, remember to map all of your bitmap entirely + areas that you want to reclaim and potentially use.

I hope you can understand what I wanted to say.

If I understood your math, you got it wrong, you shouldn't divide by 32. So if you use a bitmap to account for max RAM of 64GiB, where each bit represents a single 4KiB physical page, then the math should be:
64*1024*1024/4 bits = 16777216 bits / 8 bits = 2 MiB. That's not a huge amount, though for a low end system with say 64MiB I'd hate to lose 2MiB for no reason. Of course if you size it dynamically then there's no reason for an arbitrary 64GiB limit either.

My main point against a single bitmap is with systems that have, say 64GiB of RAM, but that RAM is dispersed over the entire 64-bit PAS (Physical Address Space). So the worst case scenario. How big bitmap do you need to be able to account for the entire 64-bit address space? Basically trying to use a single bitmap (efficiently) gets hard fast, and provides no real benefits, not even simplicity.

If you are going to use multiple bitmaps (each with different base) so as to not waste a massive bitmap to cover the potentially 64-bit PAS, then why not go all the way and have each bitmap sized zero and just record the base address of each 4KiB page, and thus you'll end up with a stack. I think a stack is actually a bit easier to implement, wastes no memory (maybe a few bytes), trivially manages any amount of RAM, etc. It easily also covers NUMA (if you're interested in that) and you can easily split the stack so each CPU (if you plan to support multi-core) can have their own stack (so no locking needed) and you can take the extra hit of locking when you want to rebalance the stacks between cores.

To put it simply, I'm not aware of any real benefit bitmap provides that stack doesn't, except the possibility of finding contiguous memory in a bitmap. But why would you need contiguous memory? The only reason I can think of is ability to use large pages, but even with a bitmap what do you think the odds are that after the system is up for a couple of hours that you'll ever again find a large page (whether it's a couple of MiB's or worse, GiB's) in a bitmap available either? I'd say pretty much zero.

Whether you both want to use bitmap is of course up to you, but presumably there's a reason your doing osdev and presumably you want to make your OS good. So try to "justify" to me or yourselves why you prefer bitmap? A stack is really easy, just hold a point to the top of the stack, store next available page there, that's it. The reason why the stack takes no memory is because the stack can give out the pages it uses when they are no longer needed for the stack. Which basically means that the stack actually does use memory, but only when the memory isn't used by anything else, once the apps request memory then the stack needs less memory, and ultimately when all memory is used the stack drops to zero bytes (or maybe the size of a single pointer).

I'd imagine a bitmap is favored because it's simple (though finding free page takes some effort, I think stack is easier, just pop the first thing off), and because it's more efficient vs a stack; stack uses full pointers (ie. 32-bit or 64-bit variables stored), but as I just explained above, it only uses memory when the memory isn't actually needed by the system, so it "virtually" uses no memory at all and is thus more efficient than a bitmap.

Osdev is all about design decisions, so why do you think a bitmap is better (aimed at both of you)?

PS. I really, really, am not trying to make you create an OS "in my image".. Just trying to offer advice. What you do with it is of course entirely up to you.

Now, i'm using Windows and if I open task manager I can see how much RAM i have...and, I want my OS to do that...


Even if I gather all length (not just the free ones) I get less with 64KiBs...I would like to get the same amount of RAM with the amount of RAM which I know that I have installed without making any approximation. By the way, how you can see in my picture, i have support for 64 bit wide.

@LtG
I'll think about your recommendation, but for now I think that I'll use multiple bitmaps because i already have the code and it supports multiple bitmaps...

Then you must be doing something wrong. I managed to count 127 MB (0x7F9DC00 bytes) of free RAM from the image you provided.

I will just quickly reply to this. Currently I don't have enough time to respond to your other question, will do it later in depth.

64 GB = 64 * 1024 MB = 64 * 1024 * 1024 KB = 64 * 1024 * 1024 * 1024 Bytes = 68719476736 Bytes conatined in 64 GB of RAM.
We don't want to store 68719476736 different addresses, we want to store blocks of memory (4 KB chunks).
4 KB = 4096 Bytes --> 68719476736 / 4096 = 16777216 Bytes/Addresses to store.
Now since we are using a bitmap there are 32 bits inside a single uint32_t, so we need to divide 16777216 by 32 = 524288 addresses to store in total.
524288 addresses = 524288 different variables = each one of them being 4 bytes so 524288 * 4 = 2097152 bytes in total to store them = exactly 2 MB.
Wow! I was missing the last line... Good catch.