CHOSTeam wrote:
Hi, I'm starting to write a basic userspace for my OS, but I do not want to use ELF or PE, I want to use my own executable format, but I do not know how to port the Clang or GCC toolchain to support it.
Can someone help me?
You probably want something like the a.out format, one of the simplest executable formats. It used to be supported by gcc/binutils and you can still probably use it (perhaps, by first recompiling gcc/binutils to enable a.out support). Linux and BSD used it before switching to ELF.
But in all seriousness, PE and ELF executables that don't dynamically link to any libraries aren't significantly more difficult to load than a.out ones, especially if you don't need relocation (a.out and PE relocation is easy, ELF is more complex w.r.t. relocation) and can load the segments/sections of the image at the address they were compiled for (PEs and ELFs w/o relocations (they're optional) are just like that: allocate memory, load, zero-out .bss, transfer control).
If you still want your own format other than what already exists, I'd not recommend messing around with gcc/binutils. It's probably going to be an overkill for you. Instead I'd recommend that you do one of the two things:
- convert ELF/PE into your format
- write a simple linker to consume the object files that the compiler produces (ELF, COFF/PE, whatever) and output an executable in your format
At any rate, loading ELF/PE or converting them into your format are about the same in terms of complexity and work. Writing a custom linker is more work.
Btw, my compiler (
Smaller C) has a linker (smlrl) that consumes ELF objects and produces a wide range of executables: flat/raw, a.out, ELF, PE, DOS .EXE. You may either try using it as-is or try to learn from it.