OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Mar 19, 2024 3:17 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 10:04 am 
Offline
Member
Member

Joined: Sat Oct 23, 2021 5:36 am
Posts: 26
I've heard people say that Windows NT disallows overcommiting memory (unlike Linux, which allows some overcommiting.) I'm confused what does it mean though. If Windows doesn't allow overcommiting memory, when does it swap pages to the disk? I always thought that memory swap occurs whenever a program tries to commit virtual memory, but there isn't enough physical memory. Correct me if I'm wrong.

Thanks


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 10:10 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Memory overcommiting is a feature that hypervisors use. It doesn't have to do with swapping, and actually is designed to limit the amount of swapping done on a server.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 10:21 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4591
Location: Chichester, UK
https://itectec.com/superuser/will-micr ... it-memory/


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 10:35 am 
Offline
Member
Member

Joined: Sat Oct 23, 2021 5:36 am
Posts: 26
iansjack wrote:
https://itectec.com/superuser/will-microsoft-windows-10-overcommit-memory/


Thanks :D


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 11:34 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5069
iansjack wrote:
(web scraper garbage)

I suggest replacing your link with the actual source.


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 4:29 pm 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 640
Location: Ukraine, Bachmut
angods wrote:
I've heard people say that Windows NT disallows overcommiting memory (unlike Linux, which allows some overcommiting.) I'm confused what does it mean though. If Windows doesn't allow overcommiting memory, when does it swap pages to the disk? I always thought that memory swap occurs whenever a program tries to commit virtual memory, but there isn't enough physical memory. Correct me if I'm wrong.

Thanks

when number of free pages hits some threshold? it then either pages out standby pages to the backing files or trims working sets. also, committing virtual memory barely can cause paging out, since never all the committed memory are really allocated (that is the mapping between physical memory established).

PS. why one would allow to commit amount of memory, one wouldn't be able to back with any physical storage, even of the second level?

_________________
ANT - NT-like OS for x64 and arm64.
efify - UEFI for a couple of boards (mips and arm). suspended due to lost of all the target park boards (russians destroyed our town).


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 4:47 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Because hypervisors tend to allocate more memory then they ever use, hence it allows lower power servers to run virtual machines more easily.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 5:52 pm 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 399
nexos wrote:
Because hypervisors tend to allocate more memory then they ever use, hence it allows lower power servers to run virtual machines more easily.


I can see virtual machine monitors overcommitting secondary storage, because mass storage is often under-utilised.

But main memory for virtual machines? It makes no sense to overcommit that.


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Fri Nov 12, 2021 6:57 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Quote:
But main memory for virtual machines? It makes no sense to overcommit that.

What do you mean? It could be very useful if someone wants to run a virtual machine with 2GB of RAM on a 4GB system with 2GB swap, while the server itself may even have to run multiple virtual machines. Then, because the VMs will likely have uneven memory usage patterns and not use most memory at once, they can all run happily.

Of course, that example is kind of lame; it would more likely be useful on big datacenter servers running hundreds of VMs on 128GB of memory.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Sat Nov 13, 2021 6:35 am 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 399
nexos wrote:
Quote:
But main memory for virtual machines? It makes no sense to overcommit that.

What do you mean? It could be very useful if someone wants to run a virtual machine with 2GB of RAM on a 4GB system with 2GB swap, while the server itself may even have to run multiple virtual machines. Then, because the VMs will likely have uneven memory usage patterns and not use most memory at once, they can all run happily.

Of course, that example is kind of lame; it would more likely be useful on big datacenter servers running hundreds of VMs on 128GB of memory.


On the contrary, your datacenter example is precisely an example of what I would not recommend.

Most OSes, tend to maximise RAM usage, as RAM is an expensive resource that is wasted if it's not used. Hence, file caches expand to use remaining unused memory. If that file cache is instead overcommitted on the underlying hypervisor machine, the memory the guest OS thinks is being put to good use caching file data is instead simply being cached (and copied) at two levels. Very sub-optimal.

If you have a 128GB RAM host memory, you're best off running the VMs with less memory and not overcommitting, and leaving it to the guest OSes to utilise that memory as they see fit.


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Sat Nov 13, 2021 7:27 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
It may not be optimal, but it certainly could keep costs down.

Also, read this aritcle

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Tue Nov 16, 2021 5:25 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 999
Overcomitting memory is certainly not a hypervisor-focused feature. It is a core memory management feature that allows virtual mappings that exceed the size of physical memory. For an actual use case, consider the address sanitizer that uses multi-TiB virtual memory mappings (for its shadow memory) that are only partially backed by physical RAM.

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Sun Nov 21, 2021 8:22 am 
Offline
Member
Member

Joined: Wed Mar 09, 2011 3:55 am
Posts: 509
angods wrote:
I've heard people say that Windows NT disallows overcommiting memory (unlike Linux, which allows some overcommiting.) I'm confused what does it mean though. If Windows doesn't allow overcommiting memory, when does it swap pages to the disk? I always thought that memory swap occurs whenever a program tries to commit virtual memory, but there isn't enough physical memory. Correct me if I'm wrong.

Thanks


Overcommitting means that when a program makes a system call to allocate memory, the OS does not return a failure status to the program even if the total amount of memory that all the programs on the system have asked to allocate exceeds the total amount of RAM plus swap on the system. The OS can do this because even when a program has allocated memory and the OS has returned a pointer to the allocated memory, that pointer just points to a location within the program's virtual address space, and the OS need not actually back that location with an actual page of RAM (or swap) until the program actually tries to access the page. This is useful because programs often allocate memory that they never touch, so not actually assigning a physical page to a virtual address until the program actually touches it prevents programs that allocate more than they need from tying up memory that other programs might need. It's controversial because the OS is telling programs that it has memory for them before it's sure that it can make good on that promise.


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Tue Nov 30, 2021 9:54 pm 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 811
Location: Hyperspace
linguofreak wrote:
Overcommitting means that when a program makes a system call to allocate memory, the OS does not return a failure status to the program even if the total amount of memory that all the programs on the system have asked to allocate exceeds the total amount of RAM plus swap on the system. The OS can do this because even when a program has allocated memory and the OS has returned a pointer to the allocated memory, that pointer just points to a location within the program's virtual address space, and the OS need not actually back that location with an actual page of RAM (or swap) until the program actually tries to access the page. This is useful because programs often allocate memory that they never touch, so not actually assigning a physical page to a virtual address until the program actually touches it prevents programs that allocate more than they need from tying up memory that other programs might need. It's controversial because the OS is telling programs that it has memory for them before it's sure that it can make good on that promise.

Yes. I recall Linux 2.0 allowed practically unlimited overcommitting with the excuse that it's impossible to predict how memory use will change in a multitasking operating system. I think they meant something like this: If process A tries to allocate more memory than is available, it shouldn't be prevented from doing so because memory may be freed by other processes before process A actually uses its memory. In practice, I saw a lot of processes killed due to segmentation faults on my 4MB 486 with 8MB swap. :) I naturally bought myself a much more powerful machine, but then an OOM-killer was implemented which, in its early form, always killed the X server. (The X server never segfaulted...)

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: "NT disallows overcommiting memory"
PostPosted: Wed Dec 01, 2021 3:09 pm 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 399
eekee wrote:
Yes. I recall Linux 2.0 allowed practically unlimited overcommitting with the excuse that it's impossible to predict how memory use will change in a multitasking operating system... In practice, I saw a lot of processes killed due to segmentation faults on my 4MB 486 with 8MB swap. :)


Ouch! By the time Linux 2.0 came out (mid-1996,) I'd graduated to a Pentium Pro with 32MB RAM. Luckily it was my work experience year at Uni, so I was actually earning, and picked up the Pentium Pro cheap at a computer fair. However, I needed a new motherboard to host it, and the only motherboard I could find was ATX based, so I needed a new case as well. That cheap PPro turned out quite expensive.

But, wow, it was fast! I think I'd come from one of these UMC monstrosities, which was slow and incompatible with earlier Linux 1.2 based distros I'd tried.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group