OSDev.org

Has anyone been able to add OS-sepcific options to GCC? For example some option like "-mgui" to mark an executable as a GUI application instead of a console application? Or perhaps adding an icon to the executable, and then my OS could recognise those specific options written into an ELF file.

Where do I start?

I see two options:
• write your own tool to modify executables after creating them (you might want to use libbfd, libelf or something similar);
• write a GCC plugin.

I would push that information into a section called something like ".glidix.info".

then i suppose the simplest solution is to create a tool which produces object files with a ".glidix.info" section in them and linking them in. But then, does the linker by default include such section in the executable, or does it remove it unless I explicitly put it in the linker script?

Why not modify the executables after linking?

Do you have two ABIs ? why would you want to differentiate something that rune in console, your compositor or someone else's?
What is a console app ? something that expect standard output or write output ?
What is a GUI app ? something that talks to your compositor ?
What about apps that so the two things , like git when it uses a GUI keyring ? Or gnuplot: no window are shown until you enter a formula in console ?
Or qemu where you tell it to output serial in console and you don't want it's GUI?

It would be useful to let the file manager determine whether it should open a terminal for a "console application" when running it. And also embedding icons so that, again, the file manager could display them.

You can also consider following macOS' approach: use app bundles. Simple executables without them would be executed with a terminal, those with app bundles wouldn't. Or you could store your options in a configuration file inside the bundle.

Well, ELF allows you to add custom sections, e.g. a resource section. Compile your resource script into an ELF object file with the resource data in a custom section and use a linker script to have ld include that section into your program executable. You could add different sections as well, e.g. certificates to prevent the OS from running unsigned programs, or metadata for COM style type libraries.

As for the console vs. GUI aspect, you could simply have different runtime startup code. Console applications can test if the standard I/O handles (stdout, stdin and stderr) are open, if not, they can create a new terminal window and bind the standard I/O handles to this window.