OSDev.org

The Place to Start for Operating System Developers
It is currently Sat Apr 20, 2024 7:14 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: "#!" implementation
PostPosted: Sat Nov 25, 2023 3:01 pm 
Offline
Member
Member

Joined: Sat Mar 28, 2015 12:50 pm
Posts: 31
I am looking at adding support for "#!" execution, and am considering the pros/cons of implementing it in kernel versus user space. There is one obvious bit of overhead doing it in user space, which is an additional system call (*). However, this alone does not seem significant enough to justify a kernel implementation.

For folks who have implemented this or know more about it, what are your thoughts? There seems to be a fair bit of variation across operating systems (this is the most detailed source of information I have found).

(*) This is assuming an implementation where we try to run the file as an executable, and if that fails, check to see if it's a script and then call the interpreter. Alternatively, we can do the check first and avoid the additional system call, but that will add overhead to launching all executables, which seems worse.


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sat Nov 25, 2023 6:57 pm 
Offline
Member
Member

Joined: Wed Mar 30, 2011 12:31 am
Posts: 676
Typically, part of the "run this file as an executable" logic is a magic check to determine both if the file is an executable and what kind of executable it is.

ELF has four magic bytes: 0x7F, 'E', 'L', 'F'. Dynamic ELF executables need to be run through a loader (conveniently called an interpreter), so you may need to load a different executable and run it in order to run an ELF.

DOS and PE executables have two: 'M', 'Z' (hence the pre-PE executables are called MZ executables).

You can implement hashbangs by treating '#', '!' as two magic bytes, and then load the requested executable just like you would a dynamic loader for an ELF.

That executable might itself be another hashbang, or an ELF with a dynamic loader, so you need to support this process being recursive - but generally with a limit (which is why execve lists ELOOP in the manual for possible errors).

_________________
toaruos on github | toaruos.org | gitlab | twitter | bim - a text editor


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sat Nov 25, 2023 8:33 pm 
Offline
Member
Member

Joined: Sat Mar 28, 2015 12:50 pm
Posts: 31
Thanks. I have previously implemented dynamic loading along the lines you described. And this is pretty much how I would add hashbang support in the kernel. But I'm wondering what is the benefit of doing that versus in libc? Is it just to avoid an additional system call, or is there some aspect to the functionality that cannot be handled in user space?

As an aside, it seems that Linux is one of the very few Unix-like operating systems that support nested hashbangs (according to this). The POSIX spec only indicates the ELOOP return code from execve for too many symbolic links in the path (not surprising since POSIX doesn't specify hashbang behavior).


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sat Nov 25, 2023 9:26 pm 
Offline
Member
Member

Joined: Wed Mar 30, 2011 12:31 am
Posts: 676
yr wrote:
But I'm wondering what is the benefit of doing that versus in libc? Is it just to avoid an additional system call, or is there some aspect to the functionality that cannot be handled in user space?

From the perspective of POSIX (which may not specify hashbangs, but it also doesn't specify binary formats at all, so a hashbang script is no less valid as a program than a compiled C binary), there is no distinction to be made here (assuming your execve is where this support exists), but you may have practical reasons to want this functionality in a kernel instead of in a libc.

Maybe you want to support suid hashbang scripts (can you do that at all in a libc? it's difficult to do securely even in a kernel), maybe you want to be able to add support for more magic interpreters in the future (do you support statically-linked binaries? one with an outdated libc might not be able to execute some newly-supported executable in the future), maybe you want to be able to add new magic interpreters at runtime (like Linux does with binfmt_misc; doing this in a way that a libc can be made aware of them would be annoying - maybe a file in /etc that gets read at startup? but what if my application is already running, do I need to re-read it on every exec?). Similar to the topic of static linking, some modern languages have opted for direct system call interfaces instead of relying on a libc - would your support patch for those also include porting the logic for hashbangs?

_________________
toaruos on github | toaruos.org | gitlab | twitter | bim - a text editor


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sat Nov 25, 2023 10:46 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1604
yr wrote:
But I'm wondering what is the benefit of doing that versus in libc?
The kernel already has to open the file to figure out what to do with it. libc doesn't. The kernel is in a way better position to figure all of this stuff out. According to POSIX, execl(), execle(), execv(), and execve() are supposed to be async-signal-safe, and that means they cannot allocate memory. But changing the command line from {"./file"} to {"sh", "./file"} requires one pointer more. You could allocate the memory on stack, but at least with execv() and execve() you risk a stack overflow in that case, especially if an attacker gets to control the argument array size.

The kernel on the other hand is in a perfect position to recognize the executable format. It has to do so anyway when asked to execute a file, and it can allocate memory for the arguments because it is in a normal syscall context.

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sun Nov 26, 2023 11:04 am 
Offline
Member
Member
User avatar

Joined: Sat Jun 25, 2016 8:29 am
Posts: 42
Location: Catalonia
yr wrote:
But I'm wondering what is the benefit of doing that versus in libc?


If I recall correctly, and following a suggestion from the #gcc IRC channel, I had to include support to execute scripts in sys_execve() (using the shebang mechanism) as it seems to be a POSIX requirement when compiling GCC natively.

_________________
https://www.fiwix.org


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sun Nov 26, 2023 1:13 pm 
Offline
Member
Member

Joined: Sat Mar 28, 2015 12:50 pm
Posts: 31
Thanks for the responses. There seems to be enough here to make a kernel hashbang implementation preferable (future enhancements, signal safety, etc.). It is interesting though that while POSIX doesn't specify anything around hashbang execution, it does say the following:
Quote:
In the cases where the other members of the exec family of functions would fail and set errno to [ENOEXEC], the execlp() and execvp() functions shall execute a command interpreter and the environment of the executed command shall be as if the process invoked the sh utility using execl() as follows:

execl(<shell path>, arg0, file, arg1, ..., (char *)0);

where <shell path> is an unspecified pathname for the sh utility, file is the process image file, and for execvp(), where arg0, arg1, and so on correspond to the values passed to execvp() in argv[0], argv[1], and so on.
This seems very much along the lines of "try to run it as an executable, and if that fails, try to interpret it as a script". This is why I was considering a hashbang implementation in user space as just a generalization of the above behavior.


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Sun Nov 26, 2023 10:34 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1604
yr wrote:
This seems very much along the lines of "try to run it as an executable, and if that fails, try to interpret it as a script". This is why I was considering a hashbang implementation in user space as just a generalization of the above behavior.
Yeah, but that only allows for shell execution. There are other script interpreters. Also, due to security concerns, there are a couple of libcs that don't implement this requirement.

Of course, these functions don't have to be signal safe, so they could also just allocate the memory. Hmmm....

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: "#!" implementation
PostPosted: Mon Nov 27, 2023 1:41 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
We could take a page from Window's book here. Have your loader support multiple executable formats in a generic manner, one being ELF and the other hashbang. E.g, we could have sys_execve actually look at a function table, with functions each format. So sys_execve would (indirectly) call elf_check_file, and if that failed call hashbang_check_file. Of course it would be looping through a function table instead of directly invoking the functions. When one succeeds, it in turn would call (e.g.) elf_load or hashbang_load, again indirectly.

Now one may think that sounds overcomplicated, but I think it's the most elegant way to handle this issue. I believe Windows operates (sort of) similarly, as it has the ability to launch .exe's, .pif's, .bat's, and so on.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 26 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group