Hi! Custom OS dev is cool! I wonder, what are your thoughts on compilers?

[ell1e]

Dear forum,

Custom OS dev is cool! Interestingly, I noticed not only does a lot of info on the wiki overlap with knowledge required to e.g. implement an assembler or other lowlevel compiler parts like a linker, but that some users also already seem to occasionally talk about compilers and custom hobby programming language projects on the forum.

Sorry for asking this as a new user but I've wondered if perhaps you ever thought about having a "Making and altering compilers & interpreters etc." section? Could be both for porting existing ones and making new ones, and talking about language projects. The reason I'm asking beyond the overlap previously mentioned, is that apparently there are barely any web forums talking about these sorts of topics in the hobbyist space.

Best regards!

[nzmjx]

Making a compiler (depending on the language) is harder than developing kernel. For instance, if you consider C++ it would require tremendous amount of developer hours to just implement the parser (because C++ have complex syntax rules and challenges; thanks to templates). Considering these, I would expect most OS developers to use either GCC or Clang (or more simpler/lightweight alternatives) on their OSes.

[Demindiro]

I believe "CompilerDev" used to be a thing, but I don't know what happened to it.

It is certainly a deep and interesting subject. I'm still looking for the exit from this rabbit hole.

FWIW (and offtopic, but) you may also be interested in https://cpudev.org/, though it is very inactive.

[ell1e]

Thank you so much for the responses, I'm glad there's some interest!

I find an OS considerably more intimidating, which is why I've worked on a compiler instead. I saw some posts about a compilerdev site, but I guess that would be an entirely different kind of difficulty to get running. But perhaps just as a subforum it might be more likely to get interest?

I'm working on two languages sharing a single compiler, one is a mix of ideas mostly from python, lua, and go, the other is some mix of ideas originating from C, zig, and swift. I'd love to see what compilers or interpreters others are working on if anybody's interested in sharing.

[sandras]

Here's my implementation of Forth.

There's not a lot of code, but I learned a lot developing it. It compiles the indirect-threaded code portion of itself. I intend to develop the machine code generating portion of the compiler in the future.

[ell1e]

Ohh that looks cool, thanks for sharing! Could I encourage you to comment the code? I don't know forth, I heard it's stack based and quite quirky but that's most that I know. And if e.g. there was some brief one-liner comment for some of the functions about their purpose, that would make it easier for newbs like me to understand your pretty-looking forth mystery better!

Also, is that assembly program a forth bytecode VM of some sort, or is it compiling to machine code?

[sandras]

Thanks for the kind words!

Right now there are only register-use (at the start of sandrasforth.S) and stack-effect (each function individually, surrounded by parentheses in Forth code) comments. I'm open to adding comments due to expressed interest. If I do, it will take time, though.

sandrasForth is pretty traditional, so if one learns Forth in general, one should be able to understand sandrasForth. I consult https://forth-standard.org a lot, but sandrasForth is not completely standardized. That's where you can read about what each function generally does, if you'd like.

Also, is that assembly program a forth bytecode VM of some sort, or is it compiling to machine code?

It does not compile to machine code. It's more like a bytecode VM, but not exactly it either. For a programmer new to Forth, I would say it basically just goes through a list (thread) of function addresses it calls. But, it is further complicated by the fact that it is an indirect-threaded Forth, as opposed to a direct-threaded one.

Like I mentioned earlier - and I'll put it another way - what you can see in sandrasforth.fth should translate directly to what is defined using, and following, DEFWORD macros in sandrasforth.S. That should aid in understanding how Forth code translates to threaded code, hopefully. And yes, sandrasforth.fth duplicates some of the code implemented in sandrasforth.S.

[ell1e]

That sounds quite cool! Does threading here refer to concurrent greenlet threads or true parallel hardware threads?

I still feel like it would be fun to have a compilers+interpreters+parsers sub forum. I recently wrote a markdown parser that I plan to overhaul again, and I felt like such projects share a lot with code compilers. I feel like such a sub forum could be fun for many people here to participate in. Unless nobody else thinks that'd be fun of course

[sandras]

That sounds quite cool! Does threading here refer to concurrent greenlet threads or true parallel hardware threads?

I think there's a misunderstanding here. I'm not referring to threads. I'm talking about threaded code. I can understand the confusion. : )

[iansjack]

If you can find a copy, the concept of TILs is well described in the classic Threaded Interpretive Languages - their design and
implementation by R.G. Loeliger. It helps if you know Z80 assembly language.

[JackScott]

I've wondered if perhaps you ever thought about having a "Making and altering compilers & interpreters etc." section? Could be both for porting existing ones and making new ones, and talking about language projects. The reason I'm asking beyond the overlap previously mentioned, is that apparently there are barely any web forums talking about these sorts of topics in the hobbyist space.

I'd be fully supportive of a "Related Topics" forum group with "Programming Language Design & Compiler Development" and "Embedded Hardware" forums. This site should always be about OS Dev first and foremost, but I think there's enough overlap in the subjects that it would be interesting to have.

[ell1e]

I would love such a subforum, especially if it was allowed for people to make a thread to report progress on one's project!

Regarding threaded interpreted languages, after skimming the book's introduction it does sound to me a little like modern bytecode VMs. Since those usually work with a bytecode representation that is usually compiled to in advance and with an operator jump table. But perhaps I'm just reading it wrong. I've never heard threaded interpreted before, I have to admit.

Although even for terms I know, I often get confused by what exactly is named what. I thought for the longest time the Lua VM was stack-based since it organizes the registers in something that it calls a stack, but since the registers can be directly numbered instead of just "use the two highest entries on the stack" I guess it counts as register-based somehow. I'll never quite understood the difference.

[Demindiro]

I figure I should show the latest interpreter I'm working on: https://codeberg.org/Demindiro/lemmings ... nterpreter

It is a Forth-like, though I'm making up words on the go. It emits machine-code directly, as it avoids the need for a separate "bytecode" interpreter. It also provides more opportunities for optimizations later, should the need arise.
I find it is easier to debug too, as gdb understands it. If I need to halt at a specific instruction, I can simply do ASM_PUSH8 0xf4 or 0xcc for hlt or int3 respectively.

I use many small word dictionaries as a form of namespaces. A namespace is simply an immediate word that reads the next word and looks it up in a specific dictionary.

It is not designed to be minimal, only simple. Hence why e.g. (Sys Door) has so many almost-identical words. I will probably make a macro to reduce duplication, though.

P.S.: The header in main.s only describes my plans, not what the interpreter can currently do. How much of it I actually implement depends on my needs.

[sandras]

Demindiro, the repository you linked to contains your Forth system, as well as code in other languages. Are you applying Forth in the context of an OS, not only implementing a bare metal Forth?

It is not designed to be minimal, only simple. Hence why e.g. (Sys Door) has so many almost-identical words. I will probably make a macro to reduce duplication, though.

We might have something common in mind. I don't try to reduce the number of assembly language primitives, but I do like to keep each primitive at just a few lines of code.

[Demindiro]

The interpreter is intended to be used as an init process which can apply various system-specific hacks as necessary, as well as launch all necessary processes.
I'm writing it in assembly as compilers don't like it when you poke random memory. By writing it in assembly I avoid any risk of misoptimizations, whether due to my own negligence or a compiler bug (the latter would be awful to debug).

I have attempted to write an OS using only an interpreter written in assembly, but I've shelved that idea for now as the x86-64 environment is very complex and I'd rather spend my time on other ideas. I'll reattempt it once I get my hands on some desktop-grade RISC-V hardware, which hopefully will be simpler.
I did make an absolutely bare-bones UEFI "Hello World" though.