OSDev.org

Hi,



If you don't think "NMI-SMI-NMI" can happen; how can you think that NMI-SMI-NMI-SMI-NMI-SMI-NMI.... can happen?

Note: Sane firmware is like unicorns - it's impossible to prove that they don't exist, and sometimes you'll see something that looks like it could be a unicorn out the corner of your eye (until you realise it's just a horse).



So, like:





Are you saying that if one measly little bit of memory that's only used by one CPU's stack and nothing else has failed, then instead of just using different memory and terminating the process, and instead of just taking that one CPU offline, I should start roaming the streets screaming "The sky is falling!" at people?



Nonsense.

If the broken RAM is in a "per NUMA domain" area of kernel space (which contains kernel code, etc) then I can take that NUMA domain offline (or maybe just replace the effected physical page/s from identical copies used by other NUMA domains in some cases). If the broken RAM is in a "per CPU" area (including a CPU's kernel stack) I can just take that CPU offline. If the broken RAM is in the message buffer area I can terminate the process that owns the message buffer. If the broken RAM was used for thread data structures or process data structures I can terminate a process. If the broken RAM was used for kernel's event log, then that can be discarded and replaced with new physical pages. If the broken RAM is in a "free physical page" stack or bitmap that'd be the easiest possible case! If ...

It's a micro-kernel, there isn't much else in kernel space.

Maybe only 1% of all memory is used for kernel space and maybe there's 1% of memory used in kernel space where broken RAM can't be recovered from; so maybe there's a 0.01% chance that an MCE caused by broken RAM is going to cause downtime because kernel couldn't recover.



It's not the tasks that matter; it's the lost data. 20 people using 20 applications on 10 computers, where each application is spread across 5 out of 10 computers - one computer dies and up to half of the users could lose data.

Yes; there's many ways to reduce/limit/mitigate that (redundant services, saving recovery information to disk in the background, etc); but preventing a computer from losing data just because someone burped is an important part of that.


Cheers,

Brendan

From what I understand, this can only happen if the SMM code somehow reenables NMIs (by executing an iret)? Otherwise the second NMI would stay blocked until the first NMI returned. If this is the case, it's not much help on consumer hardware, but for someone who controlled SMM it means things are fixable.

Hi,



Yes (maybe).

It can happen if:

• The SMM code does IRET for any reason (whether it's to intentionally enable NMI in SMM or not). Documented by both Intel and AMD
• The CPU is a Pentium processor and the SMM code invokes a trap or fault handler (without ever doing any IRET). Documented by Intel
• The SMM code clears the "NMI Mask" bit in the SMM state save map on (some?) AMD CPUs for whatever reason (firmware bugs)

This doesn't include 80x86 CPUs from other vendors (VIA, SiS, etc) where there's no documentation and no way to guess what they thought "correct behaviour" should be; and any (documented or undocumented) CPU errata that I'm not aware of.

Of course it changes nothing - there is no way for an OS to guess what the SMM code does for any computer, so the OS has to rely on "safe assumptions" (which means, OS must to assume SMM code does unblock NMI).


Cheers,

Brendan

There is a difference between broken firmware and batshit insane firmware. I'm not assuming the firmware is not broken, I'm just assuming that it is not batshit crazy i.e. that it does not re-enable NMI and then leave NMI enabled after RSM.

Of course NMI-SMI-NMI can happen iff NMI-SMI-NMI-SMI-... can happen. Note that NMI-SMI-NMI-SMI-... is not the only scenario that leads to extensive kernel stack nesting without IST. In complex kernels something like SYSCALL-#BP-IRQ-#PF-#BP-MCE-#BP-NMI-#BP might happen too. Sure, you can mitigate that by enforcing rules like "no #DB or #BP in the kernel", "no #DB or #BP in interrupt paths" or "no #PF in the kernel". But IMHO the sane option is not to ban these (otherwise completely valid) features but to assume that the firmware is not batshit crazy.

MCE and NMI on non-IST stacks is broken if you're using SYSCALL (because those handlers might actually execute on the user-mode stack). Besides that point, using the IST also gives you a safe way to access per-CPU data (by storing a pointer to the per-CPU context relative to the stack base) that works even in the presence of SWAPGS. Keep in mind that you do not even know if FS is sane in MCE and NMI handlers.

You're basically saying: Ban SYSCALL, ban #DB and #BP in kernel code, accept large kernel stacks, accept ugly and slow workarounds for FS in hot IRQ paths because there might be some batshit insane firmware (that no one of the Linux guys actually encountered yet) that leaves NMI active after SMI even though Intel actively warns against that. I'm not willing to make that trade off in my OS.


Yes. However the point is not that the actual_MCE_handler does not use IST but that it is controlled by a sane priority/nesting mechanism (the local APIC). The handler might as well run on an IST too. I acknowledge that it is ugly, but we as OS developers have no control over stupid CPU design. We have to accept that the MCIP (and NMI) mechanism is broken and work around that. Considering that this is a rare code path I'm willing to accept that ugly (but simple) code.


Basically yes, if that little bit of memory is on the MCE IST stack. Note that your no-IST MCE stack approach just tripple-faults if there is an MCE on the top of any kernel stack.

Hi,



Intel's manual (and AMD's) warn OS developers that firmware/SMM may do this, but don't discourage firmware developers from doing it (and instead provide information to help them do it). That is like an official blessing from Intel (and AMD). Maybe there are reasons for this to be necessary (I don't know enough about the SMM code in typical firmware to guess if there are valid reasons why firmware *must* enable NMI in SMM).



Sure - most OSs end up having to deal with "many interrupt sources (that have nothing to do with NMI) can nest". My latest approach is "(relatively large) per CPU kernel stack". Linux uses "small per task kernel stack; plus large per CPU stack".

Given that an OS typically has to deal with "many interrupt sources (that have nothing to do with NMI) can nest", dealing with "many interrupt sources (that can include NMI) can nest" is virtually free.



From my perspective it's the opposite: SYSCALL is broken (if you're using MCE and NMI on non-IST stacks).

For history:

• In 1997, both Intel and AMD introduced their own "annoyingly different" fast system call instructions. Intel's SYSENTER was sane. AMD's SYSENTER was broken by design (because unlike every other thing that could cause a switch to a higher privilege level it didn't load a safe ESP) and caused extra work for the kernel (having to find and load ESP itself); either deliberately (to avoid being compatible with Intel) or for marketing reasons (to pretend their SYSCALL is faster by ignoring the added cost of kernel fixing up ESP itself).
• Between 1997 and ~2000; both Intel and AMD where "annoyingly uncooperative" and refused to support each other's alternatives for the sake of compatibility, possibly because Intel didn't want to support something that's broken by design, and possibly because AMD didn't want to admit that SYSCALL was broken by design. Fortunately almost nobody ever used SYSCALL in protected mode.
• Intel tried to force people that wanted 64-bit to use Itanium (possibly to prevent future competition with AMD), and in ~2000 AMD introduced long mode (which was smart and successful). AMD also (ab)used the introduction of long mode as a way to force software developers (Microsoft) into adopting their broken by design SYSCALL instruction.
• Since then; Intel continue to refuse to support SYSCALL in 32-bit, and AMD continues to refuse to support SYSENTER in 64-bit.

This is why we are stuck with a broken by design SYSCALL instruction - due to marketing and petty rivalry between AMD and Intel, and not because of technical merit.



You're putting words in my mouth, then exaggerating fiction.

Is there any proof that none of the Linux guys have encountered an "NMI-SMI-NMI" problem? Is there any proof that the Linux guys would know if an "NMI-SMI-NMI" problem was encountered?



I'd rather do "MCE_handler: { save stuff; clear MCIP; process critical stuff; enable IRQs; process non-critical stuff; IRET; }" without extra mess and without problems.



No, due to "write-back" it would take a faulty CPU cache for this to happen; and I doubt it's reasonable to expect any OS to recover from faulty CPU cache.


Cheers,

Brendan

Ah, glad I read that correctly. The fact that it's possible for SMM code to be written to avoid this issue is still useful, in my opinion.

Someone in a position like Apple could easily take control over the firmware and write it themselves. Someone like Microsoft could include an SMM audit in their hardware verification program. Someone who's actually using x86 in a human-life-critical system (who is first of all probably slightly nuts) could of course do the same things.

It's unfortunate that the firmware has the power to cause this problem, but for a hobbyist OS it's not really an issue, and for real-world use it's at least manageable. (Whether it actually is managed is a different question.)

I looked it up in the manuals again. Intel clearly states

AMD states

That is, both manuals state that SMI-NMI-SMI cannot happen, not even for broken firmware. Intel warns that a second NMI can be nested inside the first NMI but in my understanding that concerns the "effects" of the NMI and not their dispatching (as it would otherwise contradict the previous paragraph). The following can happen: NMI -> SMI -> NMI is latched -> RSM -> NMI handler reads NMI_SC (or an equivalent register on non-Intel chipsets), deasserts the NMI sources -> IRET -> NMI is dispatched, with no NMI sources asserted in NMI_SC. The following cannot happen: NMI -> SMI -> NMI is latched -> RSM -> NMI is dispatched.


Basically this is about how much stack space you're willing to reserve.


Yes. SYSCALL is broken. NMI is also broken: Because it is unblocked by IRET and not by some NMI_IRET instruction, it causes nesting problems with exceptions like PF or BP. MCE is broken for the same reason. In a perfect world there would be a special SIRET imm8 instruction with the imm8 encoding what exceptional interrupt sources are unblocked by the IRET. However we do not live in a perfect world.

Hi,



Yes, and Intel also clearly states:



If I say "I can fly (but in special cases including reality I can't fly)" and you ignore the special case, it sounds really nice.



In the virtualisation stuff ("15.20 Event Injection") AMD also says:

..which means that if (none? some? most? all? - AMD's manual is missing a lot of detail that Intel's manual provides) AMD CPUs don't have NMI-SMI-NMI there's still potential for "nested NMI if you're running in a VM".



Yes (and I'd add SMM to that list of broken things).


Cheers,

Brendan