This is an generic discussion of how to distribute IRQ load in an SMP system.http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
talk about Linux, but there might be some useful tidbits in there, especally on the subject whether it's wise to spread out IRQs equally:
My point is that round-robin style interrupt delivery can be quite nasty on performance. It is much better to deliver interrupts from a certain device to a given core.
OK, that was an interesting viewpoint from the Linux community.
This issue seems to be somewhat related to how the scheduler works. In Linux, it seems like the device that handles the interrupt is pegged to a particular core, and thus it would be optimal to deliver the IRQ to the same core in order to minimize latency (or send an IPI to it so it grabs the server thread). In RDOS (and possibly other OSes?), the server thread is not blocked on a particular core, but rather as soon as the IRQ signals it to wakeup, it will be run by the CPU waking it up. Therefore, there is no need to peg the server thread and IRQ to a particular core in RDOS, and if the IRQ is moved to a new CPU, so is the server thread. This is also why moving IRQs can achieve load balancing.
The same goes for wakeups from timer IRQs. If a global timer is used (HPET), all timeouts will be served by the CPU that is programmed to handle the HPET IRQ, and if these are many, it would affect load balance. By moving HPET between CPUs (slowly), a balanced load can be achieved.
They also confirm my finding:
On some computers IO-APIC does not support logical delivery mode. This can be because of buggy BIOS or too many CPUs. On such computers physical interrupt delivery mode is the only thing that works, so binding single interrupt to single core is the only choice and the only thing you can do is switch the core from one to another.