OSDev.org

I've been doing some testing this morning and thought someone might be interested in the results. My machine is a Core2 Quad at 2.83 GHz. I have a 'null' system call that I used to do the tests in long mode. It does the following.

i) begins in the C lib passes 3 parameters to the kernel
ii) enters the kernel via syscall or int $0x20
iii) kernel saves all 16 GP regs
iv) uses swapgs to retrieve kernel stack, core id etc.
v) switches to kernel stack
vi) moves into kernel C code through my kernel system call mechanism and increments a counter
vii) reverses the above and returns to user mode.

The scheduler is not called.

Average times are as follows


What this means is that per call -


The other thing that this shows is that you can mix and match between the 2 mechanisms. You can use sysret to return from hardware interrupts to ring3. I also tested using syscall from ring 1 and that works fine as long as you use iretq to return.

Here is my result, test with QEMU.
The OS has only launched single process (kthreads are idle priority so will not be switched, but PIC timer will still fire which should affect very little)
The kernel and application is compiled with -O2
Note that my syscall will only preserve registers according to AMD ABI, but not 16 like yours.



Related code:


userland interface:


syscall in kernel:


null call is a C function with just return 0;

This is a lot more than what I measured on this processor in 32-bit mode with call gates in RDOS.

Results from the other thread:

near: 51.6 million calls per second
gate: 13.4 million calls per second
sysenter: 10.5 million calls per second

The call gate version takes about 75ns. I haven't measured the sysenter/sysexit version yet (edit: just below 100ns, and thus slower). And this is the full overhead as there is nothing else involved in calling kernel functions (other than loading the appropriate registers for the call in case the function has parameters). At the kernel side, no registers are saved unless they are used.

Its not reliable to use QEMU for these kind of performance tests. You must do them on real hardware.

Additionally, you should not use idealised code, but you should rather compile it and validate syscalls like you would do in a production release of your OS/application.

I just started porting my OS to 64bit last week and it just finally worked yesterday, so what I can do is run it in QEMU for now. sure someday I'll try it on real hardware.
The numbers are for references only, don't take it too serious.

oops that was mistake, should be:

The idea is to make sure rdtsc is not executed out of order.

Then, the result become:

Good testing, but I think your results were influenced by this. One of the advantage of using syscall is no need for saving all the registers. You only have to save rcx and r11. Gives considerable performance boost.

Here's how I do it. KMEM_userspace points to current TCB, which happens to be TSS as well. This is the prologue:

And this is the epilogue:

Maybe yield is interesting too:

Hope it was useful.

I understand the comments but the purpose of the test was to compare "syscall/sysret" to "int/iretq". Most documentation tells us that the former pair is "4 times quicker" (or something similar) to the latter. I've always felt that this is a useless way of comparing the instructions unless you know how much the rest of the system call costs.

As turdus points out there is another saving with sys call/sysret (i.e. not really needing to save the GP regs on a syscall). But surely this is true for the int/iretq situation as well ?

Hi,



I've always thought that, because syscall/sysret doesn't do some things that are likely to be necessary (like switching ESP to a kernel stack), it isn't directly comparable to software interrupts or call gates (or SYSENTER) because a kernel typically needs to add more instructions to a syscall handler that wouldn't have been necessary for software interrupts or call gates.

For an extreme example; because ESP/RSP isn't switched and the CPU doesn't push anything on the stack while at CPL=3, user space code could do "mov rsp, SOMEWHERE_IN_KERNEL_SPACE" and then "SYSCALL" and trick the kernel into trashing itself or modifying kernel data. To guard against that, the kernel has to save RSP somewhere and load RSP with a "known good" value before anything is pushed on the stack (either by the SYSCALL handler itself or by the CPU if an NMI or machine check exception occurs).

Note: To be honest, I'm not even sure if it's possible to use SYSCALL in a "guaranteed 100.0000% safe" way (as you can't prevent NMI or machine check before the SYSCALL handler switches to a safe stack, and task switching and IST fails for nesting).

For worst case, you'd need to deal with malicious user space code that does something like this:





Yes.

For a fair comparison that isn't overly effected by OS design, you'd want to compare:

• software interrupt with nothing more than IRET
• call gate with nothing more than RETF
• SYSENTER with nothing more than SYSEXIT
• the minimum safe SYSCALL handler

The result won't be entirely OS neutral as different OSs will take different approaches for the minimum safe SYSCALL handler (e.g. the "mov esp, *something*" part).

I'd also suggest that the caller's code size also be taken into account. SYSCALL and "INT n" both cost 2 bytes. For SYSENTER, the caller needs to store "return EIP" and "return ESP" somewhere (likely EDX and ECX), so even though SYSENTER is only 2 bytes itself it's probably going to cost 6 or more bytes. For 32-bit code, call gates are going to cost a minimum of 6 bytes (using a 16-bit address size override prefix to avoid the full 32-bit offset that's ignored anyway).

I'd expect that SYSENTER would end up being the winner for performance (for frequently executed pieces of code), and SYSCALL and software interrupts would tie for code size (for infrequently used code).


Cheers,

Brendan

If I understand correctly, in long mode (hence required by syscall instruction) ds, es are practically ignored. I do the above in my code and it affect nothing.
I still need to check cmp rsp, APPADDR_PROCESS_STACK, where APPADDR_PROCESS_STACK is the application legal address range, and have enough room, and return fail for the syscall or abort the process. syscall handler can reuse the application's user stack just fine, while make sure for not leaving sensitive data there - at that time you may still switch stack.

Hi,



The example was for a 32-bit protected mode kernel (otherwise it would've been "mov rsp,SOMEWHERE_IN_KERNEL_SPACE" ).



If the syscall handler reuses the application's user stack, be very careful with your page fault handler. If the syscall handler's RSP (inherited from user space) ends up pointing to a "not present" page (either because that's where the caller left it, or because the kernel pushed enough on the stack to cross from a present page into a not present page), then the CPU won't try to switch to a different stack when trying to start the page fault exception handler (no privilege level transition) and will generate a double fault. To avoid that you'd probably need to use IST for the page fault handler (and ensure that page faults never nest), or use IST for the double fault handler.

Also, "cmp rsp, APPADDR_PROCESS_STACK" isn't enough. Consider:



Cheers,

Brendan

according to intel manual, syscall in 32-bit or compatibility mode trigger #UD.


That's why I said "is the application legal address range, and have enough room".

edit: I did an experiment with this:


And this is catched by #PF within syscall handler, which I have a chance to terminate this abnormal process.


By the way, you are correct on the #PF issue which I overlooked.

I don't remember the parameters for SYSCALL, but at least for SYSENTER it is possible to make 100% certain that an application cannot modify kernel data or malfunctions because of an invalid kernel stack.

I do it like this:

1. Kernel ESP MSR is loaded with the current thread stack offset from TSS (by taking base + size of SS0) whenever a new thread is scheduled. This takes care of the nesting issue as ESP is not loaded manually in kernel.

2. When using the stack reference from the application, address it with the ds or es register, and let ds and es for applications only cover application address space. This will make the sysenter entry-point code protection fault if a stack reference to kernel space is provided. In long mode, this doesn't work (limits are not used), and so the pointer needs to be checked with software.

SYSCALL/SYSRET are from AMD, which does support them in 32-bit mode. Intel only supports them in 64-bit mode.

Generally if you were to use SYSCALL in long mode you simply swapgs and load in a known good pointer.



This is making the assumption you could at least clobber RAX initially as you'll probably return some value in it later. You could also do similar things for protected mode.



Here the user space GS value is assumed to be determinable from some other structure (thread info for example), which should work out since it's usually used for thread specific data anyways. To Brendan's point about NMI's it'd be a mess if you aren't using a task gate/IST for them. AFAIK Linux deals with NMI nesting by using task gates or ISTs and doing some extensive checking in software to determine if an NMI nested within the NMI handler code itself.