OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 7:48 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 4 posts ] 
Author Message
 Post subject: How to handle interrupt routing for hot-(un)plugged CPUs?
PostPosted: Mon Mar 12, 2018 4:40 am 
Offline

Joined: Tue Dec 12, 2017 6:51 am
Posts: 23
Developing my IOAPIC driver, I have a table of IOAPIC input-signals that hold the ISRs for each device IRQ. I could circularly assign each IRQ to all CPUs, and when all CPUs have been assigned, go to the next local IRQ in each CPU.

Code:

CPU      LocalVector     IOAPICInput

0          32                   0
1          32                   1
0          33                   2
1          33                   3
0          34                   4
1          34                   5

and so on



Now what is hitting hard my brain is that what would software do when one CPU is removed while the system is running? Should all IOAPICInputs be reassigned - that sounds simple but inefficient. Or is that the best solution?


Top
 Profile  
 
 Post subject: Re: How to handle interrupt routing for hot-(un)plugged CPUs
PostPosted: Mon Mar 12, 2018 8:51 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7612
Location: Germany
Uh... what kind of hardware are you looking at that supports CPU hot unplugging in the first place?

Neither Windows nor Linux do support hot unplugging, so my guess would be no-one is selling x86-based hardware prepared for it either.

So you're looking at...?

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: How to handle interrupt routing for hot-(un)plugged CPUs
PostPosted: Mon Mar 12, 2018 9:48 am 
Offline
Member
Member

Joined: Mon Mar 25, 2013 7:01 pm
Posts: 5100
ShukantPal wrote:
Should all IOAPICInputs be reassigned - that sounds simple but inefficient. Or is that the best solution?

Hot unplugging a CPU sounds like a pretty rare event. As long as you don't lose IRQs or delay them too much, you should be fine.

A better optimization might be changing your IRQ assignment strategy, since your circular method can end up steering a device's IRQ away from the nearest CPU.

Solar wrote:

Windows might not, but Linux does.


Top
 Profile  
 
 Post subject: Re: How to handle interrupt routing for hot-(un)plugged CPUs
PostPosted: Mon Mar 12, 2018 10:33 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

ShukantPal wrote:
Developing my IOAPIC driver, I have a table of IOAPIC input-signals that hold the ISRs for each device IRQ. I could circularly assign each IRQ to all CPUs, and when all CPUs have been assigned, go to the next local IRQ in each CPU.

Code:

CPU      LocalVector     IOAPICInput

0          32                   0
1          32                   1
0          33                   2
1          33                   3
0          34                   4
1          34                   5

and so on


That looks "less than ideal" for multiple reasons. The first reason is that (for APICs) the "LocalVector" part determines the IRQs priority, so you risk having (e.g.) a high speed network card having to wait for a "higher priority" ancient/slow floppy disk's IRQ handler to finish. The next reason is MSI, where you may need to assign a range of interrupt vectors to a single device. What this adds up to is some kind of dynamic "interrupt vector allocator".

The third problem is IRQ balancing - e.g. if IO APIC inputs #1 and #3 are both generating lots of interrupts then you'd want to consider sending them to different CPU cores. This should include power management (e.g. if a core is in a sleep state then send IRQs to a different CPU to avoid waking it up, and avoid higher IRQ latency and avoid increased power consumption).

Of course if you support rebalancing IRQs for (relatively common) power management reasons, then it shouldn't be much extra work to use the same code to rebalance IRQs for (relatively rare) hot-(un)plugged CPUs.

Solar wrote:
Uh... what kind of hardware are you looking at that supports CPU hot unplugging in the first place?


In general, for real hardware I think "soft offline" is more useful, especially for hyper-threading (e.g. taking one logical CPU offline so that the other logical CPU gets full use of a core). Beyond that it's supported in a few virtual machines (VVMware, VirtualBox), possibly for the same reason "memory ballooning" exists.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 4 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot] and 63 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group