kzinti wrote:
Now the solution to this without SMP is pretty straightforward: you disable interrupts between checking the condition and suspending the thread.
How do you handle this with SMP though? Well use a spinlock I suppose... So basically you need to lock/unlock the spinlock like you would disable/enable interrupts in the non-SMP case.
So how can I accomplish that? What I think needs to happen is this: you lock the spinlock, you call Suspend(reason, spinlock) and after the thread switch (i.e. sched_schedule()/sched_switch()), you unlock that spinlock. This way no other thread running on some other CPU can release the mutex before your thread is probably put to sleep.
How do you pass the spinlock to the other thread? I am thinking that it will need to be stored in a per-cpu data structure (I have one that I access with the fs or gs register). So basically:
1) Add a spinlock inside the mutex class
2) When unlocking the mutex: lock the spinlock, wakeup a waiting thread if any, unlock the spinlock
3) When locking the mutex: lock the spinlock and try to get the mutex
3a) if that succeeds, great, unlock the spinlock and we are done
3b) if that fails, call suspend with a reference/pointer to the spinlock. Save the spinlock pointer in the per-cpu data structure, sched_schedule()/sched_switch(), and one of the first thing to do when resuming the new thread is check if there was a spinlock held by the previous thread, and if so unlock it.
Sounds like this would work... But I was wondering if people had some feedback / other ideas.
Rather than a bare spinlock, my lowest level primitive is a spinlock based
monitor.
Basically, when you "enter" the monitor, you lock the spinlock (while also disabling interrupts).
While in the monitor, you can leave (unlocking the spinlock), signal the monitor (waking up other threads waiting for this monitor) or wait for a signal (going to sleep while also releasing the spinlock.)
So my monitor structure is:
Code:
struct interrupt_monitor_t {
spinlock_t spin;
thread_t owner; // For debugging and deadlock detection
thread_queue_t waiting;
};
The act of waiting is an atomic sleep and spin lock release, allowing other threads to now enter the monitor. No need to pass spinlock ownership between threads, and no lost wakeups. In pseudo code:
Code:
void interrupt_monitor_wait(interrupt_monitor_t * monitor)
{
add current thread to monitor->waiting;
interrupt_monitor_leave(monitor);
// Unlocked at this point, and the current thread is asleep on monitor->queue
reschedule();
// Another thread has signaled us from monitor->waiting
interrupt_monitor_enter(monitor);
}
Being spinlock based with interrupts disabled, it's SMP and interrupt safe, and is the primary means of synchronization between device drivers and their interrupt handlers.
It's no heavier weight than regular spinlocks in the common case of just entering/leaving the monitor, and the monitor can be used to build higher level objects. I use it to build a higher level monitor that doesn't keep the spinlock held, interrupts disabled and handles recursive use, and a reader/writer lock.
Conveniently, as it also handles waiting processes, it is also in fact the only structure I use to manage waiting processes. Everything that is sleeping is waiting on one of these structures, else it is on the run queue.
You might use it something like:
Code:
interrupt_monitor_enter(monitor);
start_device();
while(status != interrupt_complete) {
interrupt_monitor_wait(monitor);
}
interrupt_monitor_leave(monitor);
In this case, the condition is waiting for an interrupt to have occurred (as indicated by status), so you'd have your interrupt handler do something like:
Code:
interrupt_monitor_enter(monitor);
status = device_get_status();
interrupt_monitor_signal(monitor);
interrupt_monitor_leave(monitor);
All my interrupt handling uses these locks to co-ordinate between driver and interrupt handler.
I don't do SMP yet, but I don't think there are any SMP holes in my current implementation (assuming my spinlock is SMP safe.)