OSDev.org

Hi,



I've never really liked segmentation; but I do still like the idea of setting "CPL=3 CS limit" during task switches in protected mode (especially for old CPUs that don't support "no execute" page protection), which is something that would work well for me because I don't have shared libraries.



For my purposes; I'd rather just not use IST (and not support SYSCALL either). For a micro-kernel (where there shouldn't be a large number of kernel API functions) it's probably just as fast to use "call gate per kernel API function" (and avoid the likely branch misprediction for "call [table+rax*8]" dispatch); and most CPUs (all Intel) support SYSENTER anyway.

Note that SYSCALL also causes awkwardness for machine check exception handler, and if you support "machine check error recovery" (and don't just halt everything when a machine check occurs) you can't (easily) use IST for machine check exceptions either. The problem here is that if the MCIP flag is set a second machine check exception causes a triple fault (which destroys "machine check error recovery"), and if you clear the MCIP flag as soon as possible (so that a second machine check exception won't cause triple fault) then the machine check exception handler needs to be re-entrant.



I'd just assume most OSs reduce the risk but aren't immune (and that Intel's advice only covers "risk reduction" and not immunity).



I very much doubt Intel means "only the SMI is handled (and the NMI is forgotten forever)" and would assume "only the SMI is handled (at this instruction boundary)". Essentially; NMI isn't discarded but remains "pending".



You definitely won't find "uop boundaries = instruction boundaries". If an instruction is split into 5 uops there will only be one instruction boundary when the last uop reaches retirement and the entire instruction (all changes from all uops) are committed to visible state together.



No, my sequence is for "SMI does not intentionally enable/unmask NMI". Please note that Intel like to use the word "disabled" when they actually mean "held pending" (e.g. the normal IRQ enable/disable flag, which does not enable/disable IRQs and only causes them to be "held pending until IF cleared").



I'd hope most SMI handlers don't intentionally enable NMI (in the same way that I hope software is never released with critical vulnerabilities, CPUs don't have errata, and unicorns actually exist ).



I think (not entirely sure); that in 16-bit code with "32-bit operand size overrides" the SIDT and LIDT instructions only effect the lowest 32 bits of the GDT base address; and that (if the OS is 64-bit) SMM code can safely use SIDT to store the lowest 32 bits of the OS's IDT base, then do whatever it likes, then use LIDT to restore the lowest 32 bits of the OS's IDT base.



I think I found it here. As far as vulnerabilities go, I'd rate this one as "extremely plausible in practice".



I don't trust that answer at all - it completely ignores the part of the manual that the original poster quoted in their question.


Cheers,

Brendan

Hi,



That would be this:




SMM dates back to 80486 (and was backported to some later 80386 CPUs intended for embedded systems). SYSCALL came later and would've been fine if SMM didn't cause nested NMI problems.



Originally NMI was used for things like "RAM parity error" (it's wasn't something that should ever be masked or discarded). Later that type of thing got shifted to "Machine Check Architecture" (a different exception that can't be masked).



From Intel's manual:

"To prevent this situation, the processor inhibits interrupts, debug exceptions, and single-step trap exceptions after either a MOV to SS instruction or a POP to SS instruction, until the instruction boundary following the next instruction is reached. All other faults may still be generated."

From this; I'd assume that neither NMI nor SMI nor machine check is disabled, so now I'm worried about potential "MOV SS then SMI/NMI/machine check then IRQ" problems.


Cheers,

Brendan

I was contemplating of not having shared libraries as such, but rather as optimization for non-critical software (ie. games, potentially parts of browsers, etc) allowing services to be mapped to the same address space. So in that way there's no real shared libs, but any process/service could be made into one for some client processes for performance.

I'd actually need to get the OS pretty much completed to see if there's even any point or if the performance gain would be negligible.

Maybe if I stopped thinking about x86_32 features I might get more coding done...


I'll likely (hopefully) create a few different syscall dispatch method, including call gates, to see the performance difference. I have no idea of the performance impact of call gates. Note, I'm not planning on similar async syscall interface as you for various reasons, so for me the syscall overhead is likely a bigger deal..

Do any AMD CPUs support SYSENTER in long mode? If not then that leaves out a pretty big part.


Haven't really thought about MCE, but I'm assuming that there's a way to handle it..

As an aside, I keep forgetting that with NMI I'm not sure if this whole thing even matters. Are there any reasonable way to continue operation after NMI instead of BSOD/reboot?

Even if a watchdog is used then it would only trigger (on my OS) because of bugs/issues (in the kernel/OS), so there's no recovery.


Then again, the issue (using the IST-swap trick) shouldn't exist, unless you consider NMI itself to be an instruction which I've seen no support for.


According to the manuals, if SMI is triggered during NMI then all further NMI's remain "held pending", during and after SMI (after RSM). The second NMI won't be triggered at RSM, not until first IRET. So unless the SMI explicitly re-enables NMI (by using IRET), NMI's will remain "disabled"/masked until the initial NMI is handled. So that "state" of NMI's blocked is preserved thru an SMI handler.


Ah, quite possible. However that would still leave a race condition where the OS's NMI handler could potentially be called during SMI which would give the NMI ring -2. And I'm guessing that it could even be exploited relatively easily. Assuming the SMI does LIDT to restore OS's IDT and then does RSM, if NMI is triggered between those two -> OS's NMI is called.


That's the one.. Interesting read, and only two pages =)

I guess that's what happens with extremely complex systems, which get added to 30 years running..


Same here, at the very least they should have given some info as to why the other part doesn't apply, instead of referring to the NMI nesting earlier in the manual. But then again, I don't understand why the manual doesn't point from the OS dev NMI nesting part to the SMM NMI nesting part, given that OS devs need to worry about it. It should be obvious that OS dev need not read SMM since it doesn't apply, but they've dropped that little nugget there with no reference from the OS dev part.

However the OP had tested it and not able to produce the documented NMI nesting..

Ah yes, forgot about that for a moment =)


But it says "other _faults_", #MCE is an abort, not a fault, and both NMI and SMI are interrupts, are they not?

I would hate to bring this up again but the stack trick should compeletely solve these problems. Please note that my first "implementation" is not important but the idea of it.

• Absolutely no unsafe instruction windows.
• Stack does not overflow. System is immune to the "SMI/NMI" storm, e.g. a true bug walking on electronics. If this or some other physical interference lasts for a few seconds, there may be "SMIs/NMIs" literally triggered one after another with no gaps.

For the latter, what "for a seconds" could mean in the CPU world? 4 KiB stacks... are we serious?

I don't think I fully understand your solution, maybe the solution was explained in earlier posts? Is it supposed to utilize IST's or what is the stack set to upon entry to the NMI handler? I'm assuming the IDT directs NMI's to the NmiHandler, not to GeneralNmiHandler..?

If it uses IST, then how does it solve the problem Brendan says exists where the first NMI never executes a single instruction but sets RIP to point to first instruction and pushes return address on stack and then the second NMI overwrites the stack with a return address pointing to the first instruction of the NMI handler?

This is the key problem for which I came up a solution. The stack and the TSS overlap so the IST entry is modified before a single instruction of the NMI handler executes.

Ah yes, it was explained in some earlier post, I remember it now, thanks.

If possible I'd prefer something less hacky, but if it's the only alternative then it should work... Haven't really thought about every aspect of it, but at least the premise sounded plausible.

Hi,



No - AMD CPUs don't support SYSENTER in 64-bit code (even if the feature flag is set in CPUID).

Intel does something "similar to opposite" - they don't support SYSCALL in 32-bit code (even if the feature flag is set in CPUID).

This stupidity means that you can't rely on the feature flags in CPUID alone, and (at a minimum) need to follow them with a vendor check (e.g. "if feature flag FOO is set && vendor == BAR"). Then there's some errata (one Intel CPU says misreports SYSENTER support). I normally have a "CPU identification" thing during boot that sorts out this kind of stuff and generates sanitised/corrected CPU information (including having separate "SYSENTER32, SYSENTER64, SYSCALL32, SYSCALL64" flags, fixing errata, etc) and then (try to) use my sanitised data instead of CPUID after that. Unfortunately there's no "disable CPUID at CPL=3" possibility (like the "disable RDTSC at CPL=3" flag) so I can't use a "trap and emulate" approach to force user-space to use sanitised/corrected information.



That depends on how you want to handle it (e.g. if you care about "second MCE at wrong time causes triple fault with no chance of logging and no chance of recovery").



Unless you know what caused it all you can really do is "kernel panic". For my project, a kernel wouldn't use NMI for anything itself, and when NMI occurs kernel would ask the motherboard driver to deal with it (hoping that motherboard driver would have some clue about why it happened and how it should be handled). Of course the motherboard driver is in user-space, so...

Note that even if you can't recover; if you care about reliability then you must also care about making sure that the user/admin can find out what went wrong ("post-mortem"). Crashing before starting "kernel panic" or crashing during "kernel panic" (and failing to give user/admin any information about why the kernel panic happened) isn't a desirable option.



Do you think you could recover if an NMI was caused by an "NMI button" that the user pressed by accident?



For fun; try to convince me that in a world were every single thing that matters (the manual, all CPUs, all other hardware, all firmware, etc) is riddled with "things that shouldn't happen" your suspiciously convenient interpretation of "instruction boundary" is a safe assumption.

You only need to look at the various notes for "instructions retired" performance monitoring counters to see that the majority of the CPU (everything after "decode") has no clue what is/isn't an instruction (and that they had to add some kind of special tag to the last micro-op of a "multi-micro-op instruction" to make it work, and that they've got that special tag wrong in various different ways in various CPUs). Once you accept that the majority of the CPU (everything after "decode") has no clue about instructions the entire concept of "instruction boundary" has to be considered an over-simplification at best. "Results committed to visible state" is the only assumption that can really be considered "safe" (if a CPU gets "results committed to visible state" wrong it'd be plagued with issues involving corrupted visible state); but even that might be considered an "overly optimistic" interpretation after you've seen some of the issues various CPUs have had (see note).

Note: One quirk is that for SYSRET (on Intel CPUs only) if the RIP being returned to isn't canonical the CPU generates a GPF with RSP set to the user-space stack. This indicates that deep in the CPU's internals "transactional properties" aren't upheld - it's not necessarily a case of "either changes are committed to visible state or changes are not committed to visible state" but can be a case of "some changes are committed to visible state while other changes are not committed to visible state".



Heh - while swimming several layers deep in "things that happen that wouldn't happen in a perfect world", you've found another layer of things that wouldn't happen in a perfect world (involving race conditions between NMI and LIDT within SMM code).

This is progress. It won't be too long before you start seeing things as a huge ball of rusty old shrapnel being held together by duct tape, chewing gum and hacks; that is being pushed forward by a thousands of assorted CPU designers, hardware manufacturers and software developers that are all are unable see all the details of the huge ball and are all just trying to avoid being sacked this week.



As the huge ball of rusty old shrapnel rolls along, those thousands of people pushing it forward add new vulnerabilities, discover old vulnerabilities, and stick more duct tape and hacks on the ball to cover up old vulnerabilities.

Note that for a "huge ball of new shiny shrapnel" nobody has had a chance to discover vulnerabilities or cover any of them up. It's the complexity (the size of the ball) that is the majority of the problem.



The OP crafted a bow and used it to shoot several arrows at the huge ball, and those arrows happened to hit something solid and bounce back. Maybe if the OP waited a little while for the ball to roll forward (e.g. next firmware update for the same computer OP tested) the arrows would've hit something else and not bounced back.


Cheers,

Brendan

Hi,



Here, you'd have to interpret "interrupts" as "interrupts that are effected by IF in EFLAGS", because it doesn't make sense to say "the processor inhibits interrupts, debug exceptions (technically a type of interrupt which has already been mentioned), and single-step trap exceptions (technically a type of interrupt which has already been mentioned)".

More specifically; I'd assume it means "processor behaves as if IF=0 and RF=1 in EFLAGS".


Cheers,

Brendan

Handling NMI on a non-IST stack is broken. If the NMI-SMI-NMI situation indeed happens often enough your stack will overflow. Even if it only happens once it requires all kernel stacks to be large enough to handle their normal control flow + possibly multiple faults + at least two NMIs. This is not practical. What is worse is that your OS is not even able to detect this and display a nice panic screen that tells the user that there is something wrong. If you handle NMIs on an IST stack, you can at least panic.

Handling MCE on a non-IST stack is even more broken: If you clear MCIP instantly nested MCEs will just trash the MCE state. MCEs are designed to be not nested. The CPU tripple faulting on a nested MCE is a feature that prevents the OS from worrying about situations in which the whole system is broken anyways. If you're unlucky enough handling MCE on a non-IST stack might just tripple fault, e.g. if the memory that contains your stack reports errors. Sure, this might even happen for IST stacks but it is much more unlikely as there is no single point (aka cache line/bank/DIMM) of failure. Besides, how are you going to handle an MCE in your MCE handler/kernel? If kernel memory is broken you're unable to recover anyways.

IMHO the only sane way to handle the situation is putting NMI on an IST stack, detect nested NMIs that trash your return stack frame (by comparing the stored IP to some "critical NMI entry code" range) and panic if such NMIs happen. If there are rapidly nested NMIs your system is likely to be broken anyways. Why compromise the entire kernel architecture just to recover from this case? I don't think there are systems that repeatedly cause NMI without any sort of acknowledgement, precisely because this would break the x86 platform. The realistic case is not NMI-SMI-NMI but NMI-SMI-acknowledge NMI by chipset driver-NMI which is handled just fine on ISTs.

You'll have to accept that there are cases (where the peripheral hardware is broken, even though the CPU is working fine) that will just tripple fault your OS without any chance to react.

Hi,



That can/should work, even for the "SMI before NMI handler executes its first instruction" case.

However; you'd need to ensure that (for multi-CPU) different CPUs use different (physical) addresses for most of the instructions (e.g. for "cmp rsp, [Tss_RIP]", "mov rsp, [Tss_RIP]", etc); which means you'd need to use one of the tricks I mentioned (per CPU interrupt handlers and IDT; or per CPU areas). With this addition; it goes a little too far beyond my "too clever in a scary way" threshold..


Cheers,

Brendan

Did anyone ever try if NMI-SMI-NMI can actually happen? The latest Intel SDM contains the wording

along with the well known NMI-SMI-NMI warning. I suspect that NMI-SMI-NMI cannot actually happen in hardware (or maybe can only happen if the SMI handler executes IRET?) and the documentation is just plain wrong/misleading in the SDM (or targeted at SMM code writers that have to handle NMI inside SMM). One observation that supports this is that mainstream OSs do not randomly tripple fault if you're using USB keyboard emulation and watchdog timers at the same time.

Hi,



No, that's very wrong.

Using IST is broken (or at least impractical to work-around properly) for the "NMI-SMI-NMI" case. Not using IST is not broken for the "NMI-SMI-NMI" case, and as an extra added bonus you can also get a free "triple fault when no progress is possible due to NMI-SMI-NMI-SMI-NMI-SMI-NMI.... storm" advantage.

Note that (with an NMI coalescing scheme) you probably only need less than 64 bytes of stack space per nested NMI. This means that for the (extremely unlikely, in a "not going to happen for thousands of computers running for thousands of years") NMI-SMI-NMI-SMI-NMI-SMI-NMI.... storm case; with 4 KiB of kernel stack you can handle NMI nested 64 deep.



If the machine check exception handler is capable of recovery in some cases; IST is completely broken because it's impossible to avoid "second MCE trashes first MCE's stack after first MCE cleared MCIP but before first MCE did IRET". Also, for this case you do want to clear MCIP as soon as possible (after pulling information out of MSRs and storing it somewhere safe, and before you bother processing any of it) to reduce the risk of triple fault ruining your ability to recover.



Not using IST means that each nested MCE uses different cache lines/memory for its stack and there is no single point of failure (e.g. if the memory used by the first MCE is bad then it may cause a second MCE, but the second MCE will not use the same bad memory as the first). IST is a single point of failure (if the first MCE uses bad memory, the 2nd, 3rd, 4th, ... will also use the same bad memory).



How about; send an IPI to other CPUs (to tell them you're doing an "emergency soft-offline"), then send "INIT IPI" to yourself (to reset CPU and put it into a "wait-for-SIPI" state); then keep the OS running (including cleaning up any mess left behind from "emergency soft-offline") using all the remaining CPUs?



Yes; if you don't care about minimising the risk of preventable failures, using IST is an easier option. For some cases (e.g. games console?) I'd even consider leaving the NMI, machine check, and double fault as "not present" in the IDT so it all just goes straight to triple fault.



No, I don't need to accept that. What I do need to accept is that for a "peer to peer distributed" OS like mine, the failure of any one computer can effect many computers; and minimising the risk of failures as much as possible, and recovering from failures as much as possible; is a necessity.


Cheers,

Brendan

As I said, I'm not sure if this can actually happen (my reading of the SDM is that is cannot for sane firmware that doesn't do SMI-IRET to enable NMI-NMI-IRET from NMI-RSM) and even if it can it is astronomically rare and the IST solution can just panic in this case.


Just self-IPI and clear MCIP inside the IPI. Besides, if there is an MCE before the IRET your kernel memory is doomed anyways.


That won't work. If your MCE happens in kernel space (and all nested MCE that we're talking about here happen in kernel space ) those other CPUs will just MCE too. The main purpose of MCE is reporting broken RAM. The fix is to disable that RAM. You cannot reliably disable kernel RAM. Yes, MCE can also report SERR and similar errors but those are even more critical and you cannot recover from them.


If you're writing your OS with a distributed environment in mind you should be even better off: Failure of a single machine should easily be recovered from. This recovery cannot (in case of a kernel space MCE) save a single machine but should instead rely on the other machine's replicating the failing machines tasks.