Is OS development in threat of extinction?

[bellezzasolo]

Oddly enough I was just thinking on similar lines.

In an effort to keep my aging brain sharp and to stave the winter blahs I thought I would take a stab at writing a simple OS for X86. Nothing fancy, just something to waste time with.

I've been a programmer for going on 50 years now. I started out on mainframes back when it was possible for a single individual to actually understand pretty much everything there was to know about the system they were working on. I've since written 4 emulators for old Univac mainframes. I even wrote a simple operating system for one of them.

The thing that strikes me most about trying to write anything on bare metal for the x86 is the sheer complexity of it all. Coding for mainframes was simple by comparison. On mainframes much of the complexity of interfacing to peripherals was hidden in the hardware. To do I/O to any device, all you had to do was set up a couple of words in RAM, execute the start I/O instruction and sit back and wait for the interrupt. Once you got the interrupt the I/O was done, either with or without error and the data was sitting there in memory. No polling of devices, no checking of status registers, no acknowledging interrupts. Easy peasy. Granted you had to know what kind of disk drive you were talking to so that you knew how many cylinders/heads/records there were but other than that it was all the same.

On the other hand, writing a disk driver for ATA devices seems to be an exercise in frustration. PATA is easy enough, if inefficient, but get beyond that and it seems to be a lot of complexity just for the sake of complexity.

Back in the mainframe days, hardware was expensive and programmers were cheap. Now it's the other way around. One wonders why the complexity that programmers have to deal with today can't be hidden in the hardware, the way it used to be. If hardware standards were clearly and concisely written and openly published there should be no reason why hardware manufacturers couldn't create devices that just dropped into existing hardware chassis and worked with existing OSes without the need to have a driver for every different brand of disk drive or whatever.

Just thinking out loud.

ATA is clunky and backwards compatible. If you look at the really modern stuff, like NVMe, it actually does go some way like you suggest. I've never had success with ATA or AHCI, but found writing a basic NVMe driver pretty easy.

xHCI for USB is tricker, but that's mainly USB stuff, the actual host controller itself is pretty manageable.

[savoita]

I've been hobby OS-devving on-and-off for about 3 years, and I must say that whenever I thought about the sheer volume of time that would be required to implement something nearing on what my normie friends would call an "Operating System", it kinda made me want to quit. Notably, as I near in on actually writing drivers and moving away from the kernel, it constantly occurs to me that all the effort I put into writing the kernel now needs to be replicated x1000 for all the drivers I need to actually boot on a modern machine, and get a usable desktop environment.

This is aside from the fact that I will have to accompany much of this development with further work in hand-rolled testing & debugging harnesses, to ensure the changes I make don't totally bork something I wasn't paying attention to.

Which is all to say, I think it's just an unbelievably daunting hobby. With traditional programming projects, you can imagine someone finding the thing you made useful. Not so with osdev. My project is COMPLETELY useless to anyone else, lol. So you're left with the small cross-section of people independently dedicated/interested in it, and generally capable enough to actually execute on it. Which is evidently a very small cross-section.

My attitude towards it all has shifted over time, but I still want my thing to be useful, so I'm considering targeting much smaller fish (embedded). FWIW, I think that space has gotten pretty big (embedded development). It's just not often described as osdev, I don't think. Even if it's pretty much the same thing.

Also, the forum (which is usually a bit of a touchstone for the hobby, I think) is running on pretty old software. I don't particularly mind it, but I find the experience generally more pleasant on the few other Discourse-powered forums I am apart of. It's especially noticeable on mobile.

[bowdrill]

While I agree that specifications are ridiculously and prohibitively expensive in their final form, fwiw I find the free edition of POSIX extremely well indexed and easy to navigate. What do you find difficult about it?

Sorry for the late reply, and thanks for your curiosity. The annotations in POSIX definitions aren't always clear about whether an old utility will remain backwards-compatible (e.g. "First released in Issue 2"). The TOG search engine does not save me the effort of visiting every utility definition to see which ones meet my criteria.

My navigation problem aside, I don't want link rot. Final copies are indexed by line items that follow the Austin Group's mailing list discussions, so I can derive a more precise timeline than I could with a list of TOG's URLs.

[If someone wants to do X, they should know Y]. The goal of osdev resources is to help teach how to design and write an operating system, not to fill in all the prereqs.

Does this choice of norms prevent me from contributing machine-specific reading lists here or in the wiki?

[bowdrill]

Notably, as I near in on actually writing drivers and moving away from the kernel, it constantly occurs to me that all the effort I put into writing the kernel now needs to be replicated x1000 for all the drivers I need to actually boot on a modern machine, and get a usable desktop environment.

Are you saying this about the minimum effort to replicate a desktop environment, or the effort needed to replicate a desktop environment like the ones we use now?

[savoita]

A desktop environment like the ones we use now. Something I can sit a non-technical friend (which is all of mine) in front of, and they'll be more or less intuitively able to do the things they want/expect.

I realize that's always been more or less out of reach since... maybe shortly after computers were invented lol. But it feels like it gets further and further from possibility with each passing year, as complexity of systems and use cases continues to expand.

[PavelChekov]

Does this choice of norms prevent me from contributing machine-specific reading lists here or in the wiki?

That's a good point; I suppose OSDev is actually probably one of the best places for that sort of thing. I chose a poor example.

[Solar]

I don't think there is a "threat". It's not as if the development of new OS' -- specifically in the hobbyist sector -- is some kind of elementary service that would be missed if it went away. It's a pastime. If it dies out, for whatever reason, well, that's just the passing of time. New things appear, old things go away.

If somebody, anybody, really wants to do it, they still can. Actually, I think the chances of being successful -- with or without the help of OSDev.org -- are better today than ever, because the tools we can use have matured significantly.

[venos]

I actually question if this is OSDev as a hobby dying. It might be less popular than it was, I have no idea.

I suspect that, rather more accurately, the face of the web is changing. OSDev.org is an old-school setup - a wiki w/ a phpBB forum. In the face of modern social media, those are dying. Used to be any random niche hobby would have a phpBB forum or two dedicated to it somewhere, and they'd be fairly active. These days, that's kinda been replaced by a combination of Bluesky/Mastadon/etc, Discord, et al. Lots of people who would've gotten involved in the community side of an online hobby also now don't (I, myself, detest social media with a passion, niche hobby forums notwithstanding).

A combination of switching to social media, and enshittification, means communities like this are passing, but I question if the hobby itself is.

[mishakov]

I actually question if this is OSDev as a hobby dying. It might be less popular than it was, I have no idea.

I suspect that, rather more accurately, the face of the web is changing. OSDev.org is an old-school setup - a wiki w/ a phpBB forum. In the face of modern social media, those are dying. Used to be any random niche hobby would have a phpBB forum or two dedicated to it somewhere, and they'd be fairly active. These days, that's kinda been replaced by a combination of Bluesky/Mastadon/etc, Discord, et al. Lots of people who would've gotten involved in the community side of an online hobby also now don't (I, myself, detest social media with a passion, niche hobby forums notwithstanding).

A combination of switching to social media, and enshittification, means communities like this are passing, but I question if the hobby itself is.

I don't visit/post here often, but because of that I believe in the contrary - hobby osdev is blooming, and hobby operating systems (designed by few/one individuals at their spare time), a lot of times developed by younger or around my age people (I'm 24) are achieving new feats every time. Just look at Managarm, or Astral, or Mintia or countless other examples, there are "mere mortals" running Firefox inside KDE with Nvidia GPU acceleration (I am myself planning to participate in development of a universal AMD GPU driver with some people), some people run Minecraft, Windows games inside Wine, using advanced interfaces (who doesn't support USB these days?) and countless examples. And before anyone mentions AI, no, it was not used for development. What does need to change is the attitude...

This might sound preposterous, but hear me out.

Currently, all the wiki content and various OS tutorials reach a certain level of OS advancement. Past that, it’s basically mandatory that you learn how to RTFM. This isn’t a problem in itself, though.

The current trend of the OS development community makes it seem like the field of OS development is rapidly expanding. However, that is simply quantity over quality.

Exhibit A: as of posting, the last post on the “OS Design and Theory” forum was a month ago. The latest post on the “OS Development” forum (where people usually ask for help) was today. Guess what the topic of the post was? It’s an easy guess. paging

Now obviously, I am not innocent in this regard at all. A couple years ago, I was no better than anyone else who asks for PAE help on this forum.

Why is this a problem? It seems that, day by day, the OS development scene is flooded with more novices than it can accommodate. Most of the senior OS developer types, the “I was there when FSF was born” types, will in the next ten years either retire, die, or cognitively decline.
These senior OS developers have been carrying the entire hobbyist scene on their backs with virtually no compensation or recognition.

The typical OS developer these days follows a typical routine. 1. Do the Bare Bones. 2. Install Limine. 3. Struggle with basic CPU and IO configuration. 4. If they make it past that hurdle, then they write a scheduler, copying UNIX kernel designs. 5. Maybe port a libc. 6. Maybe port gcc. 7. Sit back and do nothing.

This is why Terry Davis, for all his faults, deserves recognition as a savant OS developer. He wrote the HolyC compiler, then wrote TempleOS completely from scratch (no bootloader or libc), and not only did it all alone, but he made an extremely unique (if not limited) design that was actually reasonably sensible.

My main point is that all of the extremely experienced OS developers are fading away, replaced by a new generation of socially incompetent, narcissistic teenagers who do not actually fully understand the concepts of OS development and will likely not be competent enough to keep the OS development community alive and healthy.

I do not intend on arguing this from a position of superiority. I simply argue this from a perspective that I perceive to be full of clarity. I am definitely no senior OS developer, and none of my work measures up to what most moderators on these forums have accomplished. Yet, I worry that, in due enough time, the knowledge and wisdom that is required to become a OS developer with the same skill level as a London cab driver will simply disappear alongside the prolifically technical minds who harbor them.

Thoughts?

I'm sure that that has been already, but what does actually seem to be dying (at least to me, as a somewhat of an outsider) is this osdev forum/wiki community. On one side, it is true that people are moving to different platforms and abandoning forums. But the second, and in my opinion bigger problem is the attitude of the people here. At this point, I am not even sure if the outdated (or borderline spreading misinformation) state of the Wiki is just lack of contributors/interest, or maybe a problem is that for some reason people here refuse to embrace new technologies...

On a bit of a side note, people talking about Terry Davis as some kind of genius is a whole different can of worms, his operating system was not sensible nor good (it was a lot of work, but it didn't have anything new), and in my opinion his mental illness is what had actually made him famous and ultimately lead to his tragic demise (which is what's sad to me).

What you said is not preposterous at all and I've been thinking about the exact same thing since the end of the 90s, when I started realizing that architectures are becoming more complex and documentation on the hardware more closed at each cycle. The level of complexity of a computer nowadays, even a small one like a mobile phone, is staggering. Things have evolved, not always for the best. Besides, a lot of the complexity nowadays stems for a (sometimes pathological) need for backwards compatibility. ARM didn't make things easier, as the architecture has adopted (and adapted) many of decisions (good and bad) of other architectures. Which, in part, makes sense, but it also has drawbacks. The x86-64, which appeared around 2002, if I remember correctly, also didn't make things easier, just added more complexity on top of an already complex device and it's all been downhill from there.

The new standards and architectures are being created for a reason, usually because the old ones can no longer meet some requirement. I think x86-64 is a great example of that, having learned on the mistakes of ia32 it had actually simplified a lot of things, for example you basically don't have to think about segmentation, a bunch of stuff (SSE, paging, basically APIC, a bunch of other extensions) have become architectural, the assembly is improved (how ia32 did not have relative addressing is beyond me), you have no virtual address space limitation, and on... Sure, the CPUs themselves have become more complex, but how is it more complex on the developer? And why are people still claiming it to be after more than 20 years?