I have been reading thru the intel manual on debugging.
As well as I found this article to give me the general theory of how a debugger can be created/ how they work.http://www.alexonlinux.com/how-debugger-works#table_of_contents
I am surprised that to step thru code the debugger has to modify each line first byte with the code 0xcc = int 3 to call the debugger handler
So in theory once the program that is to be debugged is loaded into memory and the debugger run on it.... the debugger modifies the code of
of the debuggie process putting that 0xcc (int 3 ) finishing that instruction and repeating to create breakpoints or stopping points in the program.
1) In linux debugging userland programs can be broken down into ptrace syscalls/int 3 and signals along with a few other things
But I am curious if you are debugging in kernel land how this all works since at that point you are no longer having a parent process and child process issue. (i.e no fork() ,...etc)
See in user land you grant gdb debugger access to debug a program thru ptrace syscalls and signals.h (i.e gdb can modify the process with 0xcc)
But in kernel land there is no gdb , or useland api so either one would have to build the functionality of a debugger into the kernels process loader
Or build a kernel level program that can be executed by the kernel process loader and have the ability to fork / have the ability to modify the child process code or something.
I am really curious about clearing this issue up.
What I am think is a kernel debugger <--(I believe that is there name for a debugger in kernel land) is that they are easier in theory because you don't have to ask permission for memory reads/writes normally since you are already in ring0 so you could in theory just have the debugger just be the same as your loader but instead of jumping to the code once loaded it would first set break points with 0xcc to maintain control of the kernel application.
In terms of security
I know bufferoverflow days are coming to an end but could one still use ptrace to allow or write a kernel module to allow him to modify a process's instructions not just placing 0xcc's for break points like debuggers do. But it could be some rude/malicous debugger program that
places machine code like ret or push some address ,..etc At this point the theory of how to write a debugger could be used to implement a malicous debugger that does this. Thus giving you the same control as buffer overflows back in the day
sssssssssshhhhhhh I won't tell