System call cost

[josecm]

Hello, I don't know if this is the correct place for this question, so feel free to move it elsewhere.

My question is related to the cost of system calls. I've always heard this is an expensive operation, but I don't really understand why. Shouldn't the cost be similar or cheaper than a common exception?

Thank you in advance

[Korona]

The exact execution times or cycles that system calls need obviously depends on the processor in question (and on the mechanism used: syscall vs. sysenter vs. int vs. call gate). However, generally system calls are quite slow. System calls take 100s or 1000s of cycles and not 10s. This cost comes from the CPU's need to reload a bunch of registers, invalidate pipelines and speculative execution state and perform privilege checks.

What is a "common exception"? If you're talking about CPU exceptions, then yes, system calls are cheaper than exceptions. If you're talking about exceptions in a high level language like C++, then that depends on how exceptions are implemented there; exceptions are usually very slow themselves.

[iansjack]

The difference being that exceptions are (relatively) exceptional, so the overhead isn't a huge problem, whereas system calls are (relatively) common.

Use sysenter (or syscall).

[Brendan]

Hi,

Hello, I don't know if this is the correct place for this question, so feel free to move it elsewhere.

My question is related to the cost of system calls. I've always heard this is an expensive operation, but I don't really understand why. Shouldn't the cost be similar or cheaper than a common exception?

A new car is cheap (compared to the cost of a new house) and a new car is expensive (compared to the cost of a sandwich). System calls are typically compared to the cost of a normal function call, and are a lot more expensive than a normal function call. The cost of an interrupt (including CPU's exceptions, IRQs, IPIs, etc) is typically a little higher than the cost of a system call.

The additional cost (for system calls and interrupts/exceptions/IRQs) can be split into 4 categories:

• The effect on the CPU's pipeline when it happens (e.g. CPU has to either discard "already in progress" instructions or wait until they retire)
• The cost of changing privilege levels (e.g. from CPL=3 to CPL=0 and back to CPL=0, including changing stacks, changing segment registers, etc)
• Bloat because your compiler isn't great (e.g. expects a specific ABI and therefore can't avoid saving/restoring various registers, etc).
• Bloat because you were lazy (e.g. an additional jump table and mispredicted branch that could've been avoided, but exists because you just did a "common exception handler" for your own convenience).

Note that avoidable bloat is typically at least as large as the actual (minimal) cost, and something like SYSCALL (that can be done in "several tens of cycles") can easily become "several hundred cycles" because of this.


Cheers,

Brendan

[obiwac]

I think this is a good topic to make a joke on...

[josecm]

Thank you for all your answers, especially Brendan who detailed exactly where this costs come from.

To obiwac,

I think this is a good topic to make a joke on...

Why? Just trying to learn...

[OSwhatever]

Hello, I don't know if this is the correct place for this question, so feel free to move it elsewhere.

My question is related to the cost of system calls. I've always heard this is an expensive operation, but I don't really understand why. Shouldn't the cost be similar or cheaper than a common exception?

Thank you in advance

Actually the answer is, system calls are always too slow. Always assume the system calls are slow. There is a lot of bulk job that has to be done before jumping to the actual function that performs the operation, like checking parameters, finding the system call function, check errors, check if reschedule is necessary and so on. This takes time compared to a function call in user space.

I've seen programs that reads the timer counter each 100us resulting a system call for obtaining the timer counter. Despite the system call is one of the faster, it is not fast enough obviously and the the program was just stealing cycles. I had to explain to the engineer that checking the time that often slows down the system and eats power. No matter how fast system calls are, assume they are slow. Avoid them if you can.

[josecm]

Hello, I don't know if this is the correct place for this question, so feel free to move it elsewhere.

My question is related to the cost of system calls. I've always heard this is an expensive operation, but I don't really understand why. Shouldn't the cost be similar or cheaper than a common exception?

Thank you in advance

Actually the answer is, system calls are always too slow. Always assume the system calls are slow. There is a lot of bulk job that has to be done before jumping to the actual function that performs the operation, like checking parameters, finding the system call function, check errors, check if reschedule is necessary and so on. This takes time compared to a function call in user space.

I've seen programs that reads the timer counter each 100us resulting a system call for obtaining the timer counter. Despite the system call is one of the faster, it is not fast enough obviously and the the program was just stealing cycles. I had to explain to the engineer that checking the time that often slows down the system and eats power. No matter how fast system calls are, assume they are slow. Avoid them if you can.

Thank you for your answer. I found your explanation very informative. However, I was actually asking the question from a hardware perspective - system call, as in the instruction itself that generates the synchronous exception.

[Korona]

I've seen programs that reads the timer counter each 100us resulting a system call for obtaining the timer counter. Despite the system call is one of the faster, it is not fast enough obviously and the the program was just stealing cycles. I had to explain to the engineer that checking the time that often slows down the system and eats power. No matter how fast system calls are, assume they are slow. Avoid them if you can.

Your general remarks about syscalls are fine; however, for the timer, that is heavily OS-dependent. Some OS (e.g. Linux, click me!) implement high-precision timers as a combination of rdtsc and a mapping that is shared with the kernel. You can call that every few us without a noticeable performance difference.

[OSwhatever]

Your general remarks about syscalls are fine; however, for the timer, that is heavily OS-dependent. Some OS (e.g. Linux, click me!) implement high-precision timers as a combination of rdtsc and a mapping that is shared with the kernel. You can call that every few us without a noticeable performance difference.

I have for a long time wanted that the HW designers of timer blocks realized this. They could put a free running counter in a separate 4kB block so that you can map it as read only into user space and just read the counter when you want. Also the user space counter could also have a dedicated divider in order to control the accuracy for the user space processes.

[zaval]

But reading the counter is not a problem by itself, polling or interrupt storm caused by such high frequency requirements are. How does your program know when "the timer event is fired"? It either gets notified through the interrupt or even worse - is polling the counter in a busy loop. Do you think that busy loop polling counter is better? Interrupts are better for sure, but still it is unavoidable to have a huge overhead with high frequency timers.

[Brendan]

Hi,

I've seen programs that reads the timer counter each 100us resulting a system call for obtaining the timer counter. Despite the system call is one of the faster, it is not fast enough obviously and the the program was just stealing cycles. I had to explain to the engineer that checking the time that often slows down the system and eats power. No matter how fast system calls are, assume they are slow. Avoid them if you can.

Your general remarks about syscalls are fine; however, for the timer, that is heavily OS-dependent. Some OS (e.g. Linux, click me!) implement high-precision timers as a combination of rdtsc and a mapping that is shared with the kernel. You can call that every few us without a noticeable performance difference.

Ironically; this creates the opposite problem - the precision you get from TSC and how quickly you can get the time is the basis for lots of different security attacks involving timing side-channels; including using cache timing to make brute force attacks on crypto libraries easier and to extract information (e.g. key-presses) from a completely "isolated" virtual machine.

Fortunately Intel knew of the security implications from the start, and added a "time stamp disable at CPL=3" flag in CR4 and have always mentioned it in their manual (something like "a secure OS should disable TSC at CPL=3 using..."). Unfortunately most OSs have cared more about performance than security and didn't follow Intel's advice.


Cheers,

Brendan

[obiwac]

To obiwac,

I think this is a good topic to make a joke on...

Why? Just trying to learn...

Nonononononono sorry I wasn't referring to your question but on the title
Which was a terrible joke in itself but anyways sorry.