Hi,
jnc100 wrote:
Brendan wrote:
During "sendMessage()" the kernel moves the entire message from the sender's message buffer to the end of the receiver's queue. During "getMessage()" the kernel moves the entire message from the start of the receiver's queue to the receiver's message buffer area.
I think I'm implementing a similar system to you. Out of interest, do you have one message buffer per process, or one per thread? Can processes/threads create buffers on demand, or is there one statically defined per process?
In my last kernel each thread had one static message buffer (created by the kernel when a thread is spawned), however my next kernel will have 2 static message buffers per thread, as I've found it's easier to write software when you can use one message buffer for sending messages while you're processing a message received in the other buffer.
jnc100 wrote:
My thinking is, if there is a single message buffer per thread, or if there is one for the process but it is only accessible by one thread, then read operations can be non-locking. Or do you simply rely on the process to only access the message buffer from one thread, as I believe windows does (?).
I'm a little unusual, in that each thread has it's own address space (and "process space" is mapped into all address spaces belonging to that process' threads). Message buffer/s exist at the same linear address in each thread's address space, and can only be accessed by that thread. This means that no read or write operations on a thread's message buffer need locks. Of course each thread's message queue does needs locks, as several threads might be trying to add a message to the same queue at the same time (possibly while the thread that owns the queue is trying to remove a message from the queue).
A more complete description of my messaging follows...
As far as normal software is concerned, when a message is sent the kernel fills from the end of the message to the end of the buffer with zeros (for security purposes) then moves (not copies) the entire message buffer from the sender's address space to the message queue, then fills the sender's message buffer with zeros. Similarly, when a message is received the kernel overwrites everything in the target message buffer with the new message (including all the zeros between the end of the received message to the end of the message buffer).
This gives a "conceptually clean" model, where the entire message buffer vanishes and re-appears in the receiver's message buffer, except for data past the end of the message which vanishes and doesn't re-appear for security purposes.
This is only how it looks though - the kernel doesn't actually do this.
Instead, (when a message is sent) if the message is "small" (e.g. less than 2048 bytes) the kernel copies the message data to the message queue, then frees all pages in the message buffer. If the message is "medium" (e.g. less than 2 MB) the kernel frees any pages that are above the end of the message, fills from the end of the message to the start of the next page with zeros, then moves page table entries onto the message queue. If the message is "large" (e.g. 2 MB or more) the kernel frees any pages that are above the end of the message, fills from the end of the message to the start of the next page with zeros, then moves page directory entries onto the message queue.
This means that after a message is sent the message buffer actually contains no pages at all (good for reducing memory usage, considering how often a thread would send a message then block until a new message is received). The kernel uses "allocation on demand" - when a thread tries to access the message buffer (after sending a message) a page full of zeros is allocated and mapped there, so that it looks like the message buffer is full of zeros as far as other software can tell (even though it's actually full of "not-present" pages). It also reduces the size of the message queues in kernel space - for e.g. a message queue might be 48 bytes long in kernel space, even though there's 4 messages on the queue that are 8 MB each (as each message would be a few page directory entries, not the data itself).
When a small message is received, the kernel frees all pages in the receiver's message buffer except the first (if present) then copies the message data from the message queue to the message buffer. When a medium message is received the kernel frees all pages in the receiver's message buffer and then replaces the page table entries with page table entries from the message queue. When a large message is received the kernel frees all pages and page tables in the receiver's message buffer and then replaces the page directory entries with page directory entries from the message queue.
To avoid hassles, the kernel's linear memory manager only allows some types of pages to exist in the message buffers. For example, a thread can't memory map a file into it's message buffer, or allocate a DMA buffer there, or map a "special mapping" there (video display memory, pages of ROM, etc). If this was possible the kernel's "sendMessage()" code would have to check for strange page types and deal with them so that the receiver doesn't receive strange pages (which would complicate the IPC code and usually be an unnecessary waste of CPU time). This means that the kernel's linear memory manager must know where the thread's message buffer/s are so it can restrict the types of pages that can be mapped in it (which is simple to do when they're at the same linear address in all address spaces).
Note: it is possible to send and receive "not-present" pages as part of a message (if the message is medium or large). For example, if a page in the message buffer was sent to swap space, then it's possible for that page to still be in swap space after the receiver receives the message. In the same way it's possible to send and receive "not-present, allocate on demand" pages that look like they're full of zeros (but actually don't exist). These "not-present" pages don't really complicate the kernel's IPC code - if the IPC code touches data in a "not-present" page then the page will be mapped in (same as if the sender or receiver tries to touch the data), and the IPC code itself doesn't really need to care.
Also, I guess I should mention that for the last version of my OS messages could be up to 32 MB, and all message buffers took up 32 MB of the address space. For the next version of my OS this will probably be reduced, but I haven't decided what the maximum size of a message will be yet (it'll be somewhere between 4 MB and 16 MB).
Cheers,
Brendan