FunnyGuy9796:
syscall is designed to be executed from usermode to switch into kernelmode
you can omit its execution in kernemode as users here suggested you already... when you run kernel no need to execute the SYSCALL instruction, you can use the CLI instruction
the only one execution of syscall in kernelmode I've ever seen was in ms windows kernel to detect hypervisors tampering with the address of MSR LSTAR (one of the many patchguard detection methods)
Code:
INITKDBG:00000001402F4470 sub_1402F4470 proc near ; CODE XREF: .text:000000014018D3DDp
INITKDBG:00000001402F4470 ; INITKDBG:00000001402E7923p
INITKDBG:00000001402F4470
INITKDBG:00000001402F4470 var_18 = dword ptr -18h
INITKDBG:00000001402F4470
INITKDBG:00000001402F4470 mov ecx, 0C0000084h
INITKDBG:00000001402F4475 rdmsr
INITKDBG:00000001402F4477 push rdx
INITKDBG:00000001402F4478 push rax
INITKDBG:00000001402F4479 and eax, 0FFFFFEFFh
INITKDBG:00000001402F447E wrmsr
INITKDBG:00000001402F4480 pushfq
INITKDBG:00000001402F4481 or [rsp+18h+var_18], 100h
INITKDBG:00000001402F4488 popfq ; enable rflags.TF so when the next instruction execution completes, #DB will be generated
INITKDBG:00000001402F4489 syscall ; #DB generated after this instruction completes, note we are in kernemode but executing syscall instruction
INITKDBG:00000001402F448B mov r10, rcx ; grab the real address where MSR LSTAR is pointing
INITKDBG:00000001402F448E mov ecx, 0C0000084h
INITKDBG:00000001402F4493 pop rax
INITKDBG:00000001402F4494 pop rdx
INITKDBG:00000001402F4495 wrmsr
INITKDBG:00000001402F4497 mov rax, r10
INITKDBG:00000001402F449A retn
INITKDBG:00000001402F449A sub_1402F4470 endp
KineticManiac:
you can't trivially disable #NMI on Intel, you can on AMD using the CLGI instruction
I spent a lot of time to solve the #NMI problem on Intel during hypervisor development. Hyper-V uses #NMI to communicate among cpus/cores. I'm developer of hypervisor with nesting capabilities and I had to solve these #NMI delivered into my parent hypervisor (which is solvable trivially by NMI window exiting and #NMI injection using VM-entry interruption-information field in most circumstances, but complicated when #NMI was delivered very late - just before vm entry). Intel has some internal CPU flag to put CPU into blocking mode for further #NMI deliveries, so the CPU handles only one #NMI at any time, the last instruction of such #NMI handler is the IRETQ instruction which unblocks #NMI so further #NMIs could be delivered. It is impossible to check/obtain this internal cpu flag on baremetal, but you can access and modify it when enabling and using VMX (VMCS, guest interruptibility state, blocked by NMI bit). But you can access/modify it only for non-root mode (guest), you still cannot access it for yourself = root mode (your hypervisor).