Hello and thanks for the answers,
Brendan wrote:
Hi,
For 80x86, you can freely mix 4 KiB, 2 MiB and 1 GiB pages within the same virtual address space.
From intel manual, it was not clear for me that I could mix different pages size in the same page directory, thanks for the clarification. That was my main question because I did not know how mix physical memory and devices mapped on memory.
Brendan wrote:
Note that CPUs have a limited number of TLB entries for various sizes. Usually there's thousands of TLB entries for 4 KiB pages and tens of TLB entries for larger sizes. This means that if you use a small number of 1 GiB pages it helps performance (less TLB misses), and if you over-use 1 GiB pages you get worse performance (lots of TLB misses because most of the TLB entries can't be used).
Don't forget that (with the recent "meltdown" vulnerability) you're going to want to be using PCID where possible, which means that the TLB has to contain entries from multiple different virtual address spaces, making "number of TLBs that OS can use" even more important.
Also, "cacheability" is stored in TLB entries too (so that the CPU doesn't have to consult MTRRs when there's a TLB hit). If you use a 1 GiB page for an area that contains a mixture of "cacheability" types (e.g. according to MTRRs, some is "write-back" RAM and some is an "uncached" memory mapped device) then the CPU may end up using many 4 KiB TLB entries instead of using a 1 GiB TLB entry so that it can store the "cacheability" in the TLB entries properly. CPU designers have special hacks to make this work for the first 2 MiB of the physical address space (with 2 MiB pages) where it's known that there's often an ugly mixture (some RAM, some memory mapped devices, some ROM) and this has probably been tested to make sure it works; but for all other areas there's a relatively high risk of subtle and not so subtle bugs in the CPU where using 1 GiB pages for "mixed cacheability" areas can lead to problems (e.g. INVLPG only invalidating 4 KiB of the 1 GiB area and causing an OS to crash in unpredictable ways).
Got it
Brendan wrote:
Finally; usually an executable has several sections with different attributes (".text", ".rodata", ".data", ".bss", ...) and the permission bits (read/write, no-execute) in page tables are configured to reflect the section's attributes to improve security and/or detect bugs; and usually an executable also uses shared libraries. This means that if you only use 1 GiB pages and nothing else, a tiny little "hello world" executable will consume about 8 GiB of RAM (1 huge 1 GiB page for each section in the executable and each section in the libraries it uses); and (based on "half a page wasted per section") on average (for "1 GiB pages only") every process will waste 4 GiB of RAM.
I understand your point, but in my case, I am not reflecting the right access to those sections by using the permission bits. The only bits I am using are the cached/uncached bits (and that was actually my main issue).
Brendan wrote:
Mostly, 1 GiB pages (and 2 MiB pages) should be used sparingly (e.g. only if a process is using the whole 1 GiB with the same permissions anyway); and failing to do that will hurt performance and/or waste a lot of RAM (which will also hurt performance - less RAM for file data caches, etc). If you only support a single page size, then it'd be much better to only use 4 KiB pages (although in this case it's usually not that hard to use larger page sizes where appropriate for memory mapped devices, because the physical memory manager doesn't have to deal with larger page sizes in this case).
I think my case is very particular. I am not heavily using paging. I have a flat memory space which is mapped one to one to the physical memory. I have only one page directory and the main idea was to speed up the time that the mmu takes to walk through the page directory. By using 1GB, I will be able to reduce the size of page directory (currently I am 2MB pages). However, I am not sure if there is really any improvement. I need to experiment a bit before.
Thanks again for these great answers, Matias.