OSDev.org

To give a more specific but still brief description of the problem: When I make any syscall from my C function - which itself is called from a userspace assembly procedure - the syscall returns to the C function, but then the C function doesn't return to the assembly procedure which called it.

Here's the code segment which calls the C function:


Here is the C function which is called:


And finally, here's a very minimal example for my syscall ISR which causes the issue:


As I step through the program to debug it in bochs, I monitor the stack. Here's what I notice:

At the beginning of the C function, the correct return address is at the top of the stack (happens to be 0x100755). So here, the stack looks like this:

(mem addr: value)
0x104ffb: 0x100755

The C function then pushed ebp (which here is 0) to the stack. It then saves esp into ebp.

0x104ff4: 0x0
0x104ffb: 0x100755

As soon as the C function calls int $0x80 (which is what I've chosen to be my syscall), four numbers are pushed to the stack:

0x104fec: 0x100764
0x104ff0: 0x1b
0x104ff4: 0x202
0x104ff8: 0x104ff4 <- this is equal to ebp... why?
...

The worrying part here is that this seems to have overwritten the 0x0 and the 0x100755 which were previously at the top of the stack. They're nowhere to be found on the stack whatsoever anymore, so, this is where I think it all went wrong.

When it returns to the C function it has popped the top two things from the stack, so the top now looks like this:

0x104ff4: 0x202
0x104ff8: 0x104ff4

And from there, it pops 0x202 into ebp (which is wrong), and then fatally, it returns to the address 0x104ff4, which is where the stack was at some point. From there, it just executed garbage memory and crashes.

So, any idea what's going on here? I'm not so familiar with using C functions with assembly, and have quite likely made a beginner's mistake here.

Thanks a lot

Did you perhaps set ESP0 in your TSS to the same value you use for your ring 3 program's initial ESP? If so, every switch to ring 0 will clobber your program's stack.

I saw a couple other issues:


This inline assembly modifies EAX without telling the compiler.


Use an interrupt gate to clear the interrupt flag at the beginning of your interrupt handler instead of CLI. (Use a trap gate if you want to set the interrupt flag.) Don't bother messing with the interrupt flag immediately before IRET, since IRET will pop the flags from the stack anyway.

Ah, yes, I was setting my ring 3 ESP to the same as my ESP0 in the TSS (I think, at least... I haven't looked at that code in a while). What's the usual solution? Reserve another bit of memory for ESP0 in the TSS?

EDIT: I've fixed it! So it was basically what you said, but just in case anyone's stuck looking at this in the future, I reserves some new memory for a second stack in .bss, and when iret'ing into userspace i told it to use this stack in ESP.

The usual solution is to put everything belonging to the kernel (in ring 0) in the highest 2GB of address space, and leave the remainder for the current user (ring 3).

But then, usually the ring 3 code isn't part of the kernel, so that may not be the best setup for you. Fortunately, anything that separates your ring 0 and ring 3 stacks will work, so you're free to do that however you like.

The only reason there's ring 3 code in the kernel here is just to test ring 3 code. Usually, ring 3 code will be loaded from the filesystem like normal, I just haven't implemented the file system yet.