OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 5:42 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: How can i improve memory management?
PostPosted: Thu Apr 26, 2018 12:02 pm 
Offline
Member
Member

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 587
Currently, I have a "frame bitmap" allocator. Each bit represents a 4KB frame, which also allows me to allocate multiple consecutive frames. I also have a disk cache and a file cache:

- The disk cache is per-disk and stores lists of tracks (32KB segments) from disks as big reads/writes are fastest. (Well, technically it's not a list, but rather a tree much like x86 page tables). Each segment is physically consecutive, to help with DMA.
- File cache is per-inode and stores pages (4KB) from files.

When the physical memory allocator cannot find any free frames in the bitmap, it asks the caches to free up some memory. There are some problems tho. Mainly, if the disk cache gets competly emptied, it's usually impossible for the files to be flushed, because the FS driver tries to write a file to disk, there is a disk cache miss, so the disk cache again asks the file cache to free up some memory, but it doesn't end up being 32KB consecutive, so it fails.

This system literally does not work: if the caches fill up RAM, it fails to free up any memory sooner or later.

What alternative, cache-aware PMM could be used instead?

Thanks.


Top
 Profile  
 
 Post subject: Re: How can i improve memory management?
PostPosted: Thu Apr 26, 2018 4:16 pm 
Offline
Member
Member

Joined: Mon Jul 05, 2010 4:15 pm
Posts: 595
Mixing page sizes like this is usually very difficult. Fragmentation can also destroy your ability to have any 32kB page available.

A few remarks, since you always require 32kB consecutive pages, could reserve a set of them so that you always have some available?

If you don't have consecutive 32kB pages, do you have the possibility to have a fall back so that you can still use storage IO even if it becomes a little bit slower.


Top
 Profile  
 
 Post subject: Re: How can i improve memory management?
PostPosted: Thu Apr 26, 2018 4:39 pm 
Offline
Member
Member

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 587
OSwhatever wrote:
Mixing page sizes like this is usually very difficult. Fragmentation can also destroy your ability to have any 32kB page available.

A few remarks, since you always require 32kB consecutive pages, could reserve a set of them so that you always have some available?

If you don't have consecutive 32kB pages, do you have the possibility to have a fall back so that you can still use storage IO even if it becomes a little bit slower.


No, I don't have a fallback. Would it be better if the disk cache was in 4KB pages, and simply using a temporary 32KB buffer for the transfers?

Also, the perfomance is terrible: searching a tree to find a page that could be freed appears to be very ,very slow...


Top
 Profile  
 
 Post subject: Re: How can i improve memory management?
PostPosted: Thu Apr 26, 2018 5:27 pm 
Offline
Member
Member

Joined: Mon Jul 05, 2010 4:15 pm
Posts: 595
mariuszp wrote:
Would it be better if the disk cache was in 4KB pages, and simply using a temporary 32KB buffer for the transfers?


You shouldn't paint yourself into a corner regarding the sector/cluster size as this size is not given by the OS but rather the storage device or the file system. There is nothing wrong having the ability to have larger cluster sizes for the swap file and several existing operating systems allow adjustable size.

Your problem is that you really need consecutive pages when you are low on memory. What system is this? Don't you have an IO MMU?


Top
 Profile  
 
 Post subject: Re: How can i improve memory management?
PostPosted: Fri Apr 27, 2018 6:46 am 
Offline
Member
Member

Joined: Sat Oct 16, 2010 3:38 pm
Posts: 587
OSwhatever wrote:
mariuszp wrote:
Would it be better if the disk cache was in 4KB pages, and simply using a temporary 32KB buffer for the transfers?


You shouldn't paint yourself into a corner regarding the sector/cluster size as this size is not given by the OS but rather the storage device or the file system. There is nothing wrong having the ability to have larger cluster sizes for the swap file and several existing operating systems allow adjustable size.

Your problem is that you really need consecutive pages when you are low on memory. What system is this? Don't you have an IO MMU?


It's on x86_64, and the allocator works in the way I described in the first post. I'm not sure what an IO MMU is. I could keep a 32KB track reserved for "emergency", but wouldn't that just make the whole system even slower?


Top
 Profile  
 
 Post subject: Re: How can i improve memory management?
PostPosted: Thu May 17, 2018 9:27 am 
Offline

Joined: Sat Jan 27, 2018 1:23 pm
Posts: 4
For consecutive physical RAM you first aproximate which devices need that and how much. Allocate that very early during system boot.

If you need other sizes of consecutive physical RAM then you have to walk all linear mem chunks and see what and how you can substitute physical RAM.
You can simply traverse paging structures. Slow but doesn't require any additional permanent data structures.
On the brigth side, devices drives usually allocate consecutive physical RAM very rarely.

I use 16KB linear chunks that map to 16KB consecutive physical RAM chunks for everything. And I have separate 4KB pool of physical RAM.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 23 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group