OSDev.org

The Place to Start for Operating System Developers
It is currently Sun Sep 20, 2020 4:45 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 11:14 am 
Offline
Member
Member

Joined: Fri Jun 02, 2017 3:01 pm
Posts: 29
Hi, I'm starting to write a basic userspace for my OS, but I do not want to use ELF or PE, I want to use my own executable format, but I do not know how to port the Clang or GCC toolchain to support it.
Can someone help me?

_________________
Developing CHicago Operating System.


Top
 Profile  
 
 Post subject: Re: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 11:34 am 
Offline
Member
Member

Joined: Thu Apr 26, 2018 11:21 pm
Posts: 49
Before that, have you written a working ELF/PE loader, e.g. for bootloader or drivers and now you want an alternate format?


Top
 Profile  
 
 Post subject: Re: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 11:51 am 
Offline
Member
Member

Joined: Fri Jun 02, 2017 3:01 pm
Posts: 29
For loading the kernel i'm using GRUB, but for the drivers and the userspace i want to use my own format.

_________________
Developing CHicago Operating System.


Top
 Profile  
 
 Post subject: Re: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 12:25 pm 
Offline
Member
Member

Joined: Thu Apr 26, 2018 11:21 pm
Posts: 49
I would suggest if you haven't, that you write a PE Loader to load a kernel driver, maybe the driver just writes a character to 0xB8000 video memory to show you it runs, because PE is definitely far easier than ELF to understand.

Another reason is because PE covers all the ground you'd want (and need) to learn before doing your own format.

If you've done that, then you have everything you need to start your own format. Essentially all an executable format is, is a way of describing how a program should be laid out in memory and where to relocate (fix up) any referenced addresses according to the memory it was loaded in.

Lastly you'll need to design your own linker to output an executable in your format. I wrote a PE linker in 2 weeks with little prior knowledge, and applied small fixes over a period of months, maybe if you have less experience the majority of code could take a month or so.

The simplest method would be to have Clang/GCC output .obj files in a known format (like ELF or COFF/PE), then your linker takes the object files and merges them into an executable in your format.

The harder method would be extending Clang/GCC's assembler to output files in your own object format then extend Clang/GCC's linker to merge (link) them all into a file of your executable format.


Top
 Profile  
 
 Post subject: Re: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 12:33 pm 
Offline
Member
Member

Joined: Fri Jun 02, 2017 3:01 pm
Posts: 29
Well, i already have the base of my executable format, and a working loader for it, i just need to create a linker for it.

_________________
Developing CHicago Operating System.


Top
 Profile  
 
 Post subject: Re: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 12:42 pm 
Offline
Member
Member

Joined: Thu Apr 26, 2018 11:21 pm
Posts: 49
Right. The easiest way to do that is take ELF/PE .obj files and write a linker to merge them into your format. If you've got a working loader it won't be much harder to do the linker, it's just reading .obj files, recording information about different .obj files, combining sections (.text, .data) into one steady stream of data and applying any references between the .obj files to the stream

The base of doing that, is having a structure in your linker that maintains a data pointer, allocated size and current (used) size, and you call functions to add data to it (and realloc if needed) on demand. Then you maintain lists of information you need to link, and where they've been copied to in the 'linked' data stream.


Top
 Profile  
 
 Post subject: Re: Custom executable format in Clang/GCC
PostPosted: Mon Jun 18, 2018 8:33 pm 
Offline
Member
Member

Joined: Tue Mar 04, 2014 5:27 am
Posts: 1025
CHOSTeam wrote:
Hi, I'm starting to write a basic userspace for my OS, but I do not want to use ELF or PE, I want to use my own executable format, but I do not know how to port the Clang or GCC toolchain to support it.
Can someone help me?


You probably want something like the a.out format, one of the simplest executable formats. It used to be supported by gcc/binutils and you can still probably use it (perhaps, by first recompiling gcc/binutils to enable a.out support). Linux and BSD used it before switching to ELF.

But in all seriousness, PE and ELF executables that don't dynamically link to any libraries aren't significantly more difficult to load than a.out ones, especially if you don't need relocation (a.out and PE relocation is easy, ELF is more complex w.r.t. relocation) and can load the segments/sections of the image at the address they were compiled for (PEs and ELFs w/o relocations (they're optional) are just like that: allocate memory, load, zero-out .bss, transfer control).

If you still want your own format other than what already exists, I'd not recommend messing around with gcc/binutils. It's probably going to be an overkill for you. Instead I'd recommend that you do one of the two things:
  • convert ELF/PE into your format
  • write a simple linker to consume the object files that the compiler produces (ELF, COFF/PE, whatever) and output an executable in your format

At any rate, loading ELF/PE or converting them into your format are about the same in terms of complexity and work. Writing a custom linker is more work.

Btw, my compiler (Smaller C) has a linker (smlrl) that consumes ELF objects and produces a wide range of executables: flat/raw, a.out, ELF, PE, DOS .EXE. You may either try using it as-is or try to learn from it.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group