OSDev.org

Hello,
So in a multitasking kernel, how do i handle multiple API calls at once? In other words, can two threads run in kernel mode side by side, or should there some lock to prevent this? If so, should it be that only one thread period can run in kernel mode at one time, or that one thread of a process can be in kernel mode at one time?
Thanks,
nexos

Are you using multiple CPUs (I don't have any experience with multiple CPUs).
Otherwise, I would think that you would be running different threads for different functions, each with their own data (loaded into memory as if they were a program). If they are both accessing the same device, yes a lock is necessary to prevent each thread from corrupting each other.

(This answer is to the best of my knowledge. However, I haven't yet implemented locks in my own operating system, so I may be wrong about something)

It's however you implement it. You only need to serialize access to some global/shared resource and that only if any modifications are ever made to its state. When there's no contention for that resource, you can have multiple threads going about their business independently. Think readers-writer lock on a database. Things become interesting when you need to have mutually exclusive access to multiple such resources and you're running a risk of deadlocks or live locks.

Design as if you can have multiple threads running at once.

If you're using a very small microkernel, that basically just passes messages around between user mode processes (and provide MMU and other privileged CPU access) you MIGHT be able to get away with a single kernel lock and serialize access to kernel mode.

But each CPU will still require it's own kernel stack, and when you send inter-processor interrupts, you'll necessarily have multiple threads running in kernel mode on different CPUs.

Linux made the mistake of having a single kernel lock (google BKL) and it took years and years to remove.

In short, don't make the same mistake.

Certainly I would make no distinction between threads in the same/different processes.

You only need to enforce mutual exclusion where it's necessary - enforcing it on the whole kernel incurs a big performance penalty, although is a lot easier than allowing multiple threads in the kernel.

Take an NVMe drive. NVMe allows you to create numerous submission/completion queues, and my OS aims to have one of these per CPU (in fact, when I add priorities, that'll become 4).
The hardware itself deals with round-robin weighted scheduling.

This means that the control path for e.g. a read request is quite simple. Usermode requests a read, specifying a virtual buffer.

The kernel then needs to verify that buffer - ensuring it's present, and is also user mode memory (thus preventing a malicious program from overwriting kernel memory). This is also the point where you'll want to lock the pages into memory. You may need to acquire an address space lock, or a lockless algorithm might work - the address space could be in use on multiple CPUs (and kernel space is in use on them all).

That's also a convenient point to build a scatter/gather list of the physical addresses involved.

Once that's done, then you can simply create the command, and add it to the submission queue. If you have a queue per CPU, then the only atomicity you need here is preventing preemption of the kernel.

Sorry for potential ignorance, but what do you mean by "then the only atomicity you need here is preventing preemption of the kernel". What does preventing preemption of the kernel mean?

I have read up on preemption, and am wondering if I could disable ints during a system call. It appears old Linux did this. Eventually, I would make it preemptive.

Yep, disabling interrupts can be done just for a critical section - in the NVMe example, that's between acquiring a command entry, and finishing writing it. However, in that example, my actual implementation uses cmpxchg to increase the pointer, after incrementing a counter of commands being written. Only when that counter is 0 will it ring the NVMe's doorbell. While that model could cause problems if there's a lot of contention, if it's just a kernel preemption, it's a more than adequate solution, and avoids disabling ints.

I've gone with that, rather than disabling ints, only really because there's no guarantee that a queue won't be shared between CPUs.

You don't necessarily need to disable interrupts during system calls to prevent preemption, you just need to defer any scheduler action resulting from an interrupt until the current syscall is completed; see the example below. Also, even a preemptible kernel will have critical sections where it disables interrupts, but most of the time it will have interrupts enabled and be ready to take scheduler action immediately when required.

Example:

1) Process makes system call, CPU enters kernel mode.
2) Kernel begins processing syscall.
3) Timer interrupt occurs, and the current process's timestep has expired...
4) ...however, the process is in the middle of a system call. Kernel makes note to itself to run the scheduler when the system call completes.
5) Kernel finishes system call, process is now ready to return to user mode.
6) Before returning to user mode, the kernel checks to see if the scheduler needs to be run.
7) Kernel finds its previous note, and calls the scheduler.
Scheduler selects a new process to run.
9) One* or more processes run.
10) The original process is eventually scheduled again.
11) The process returns to user mode.

On a system with a preemptible kernel, the steps would run in the order 1, 2, 3, 8, 9, 10, 5, 11. Steps 4, 6, and 7 wouldn't happen at all.

*Actually, for step 9, there is potentially the case that the current process is the only one in a runnable state (such as if all other processes are waiting on I/O), in this case the scheduler might just select the same process again, so we can really say "Zero or more processes".