With my queues, I want to consumer to sleep and wait for messages, so I had to invent a system call - sleep_if_not_set(size_t address) and it sends the thread to sleep if the address at that memory location is not set.
However, the difficult part for me is that many of my interrupt handles (such as the keyboard and mouse handler) write to these queues, system calls like "open_file" send the VFS messages, etc, and I worry about sharing a mutex between a thread and an interrupt handler because if an interrupt handler locks a mutex locked by a thread I'll end up with a non-returning deadlock (for example - my Window Manager locks the queue to grab the top event, meanwhile the mouse moves and an interrupt fires, and the interrupt handler encounters the locked WM queue.)
My temporary solution right now is to temporarily disable interrupts when touching data structures that can also be touched by interrupts:
Code:
disable_interrupts();
/* read from queue */
enable_interrupts();
My interrupt handlers don't multitask, so I could put this code inside of two system calls - "Push to Queue" and "Pop from Queue" - which would make the operations atomic on a single core system, but this achieves the exact same effect as wrapping my pushing/popping in cli/sti but with the overhead of a system call.
It works for now, but it's going to break when I implement SMP, unless my kernel mutexes both disable interrupts and use spinlocks. All of this seems inefficient for a simple message queue data structure. Am I heading down the wrong path?