OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Mar 19, 2024 5:01 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 34 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 7:04 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 999
I think I've written about this before, but: frequent iteration should always be prioritized over coming up with a great design from scratch. Of course, every once in a while, one should sit down and think about design and whether one's current design is on the right track. However, without frequent iteration and simply trying out things it is not possible to understand the limitations and difficulties of different designs. It is also important to accept that in any project, a large portion of the code is not in an optimal shape. That is not because programmers are lazy or because programmers are stupid (especially the latter is of course not true), it is just a consequence of the fact that it's impossible to understand all requirements when coding from scratch.

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 7:09 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
IMO language doesn't matter too much, I think using it good ol' plain C and bits of ASM is good. My main goal is speed, security, and stability. These may seem obvious, but stability / security and speed appear to be mutually exclusive. I want to try to mix them together as best as possible. Oh, I'll remember to read the Intel manuals before writing code :wink: .

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 7:11 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Korona wrote:
I think I've written about this before, but: frequent iteration should always be prioritized over coming up with a great design from scratch. Of course, every once in a while, one should sit down and think about design and whether one's current design is on the right track. However, without frequent iteration and simply trying out things it is not possible to understand the limitations and difficulties of different designs. It is also important to accept that in any project, a large portion of the code is not in an optimal shape. That is not because programmers are lazy or because programmers are stupid (especially the latter is of course not true), it is just a consequence of the fact that it's impossible to understand all requirements when coding from scratch.

Exactly. I think I've done enough research into OSes and made enough research OSes to where I have the ability to make something good.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 8:00 am 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 811
Location: Hyperspace
@Korona: I agree; I've (finally, after much protesting ;) ) come to the same conclusions.

@nexos: Some languages are special, Lisp and Forth especially so. You can code an ordinary OS in these languages, perhaps with more effort, but being interactive and having simple syntaxes, I think they can enable scripting components together in a way which puts Unix to shame.

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 12:16 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out :) .

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 2:20 pm 
Offline
Member
Member
User avatar

Joined: Tue Sep 15, 2020 8:07 am
Posts: 264
Location: London, UK
nexos wrote:
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out :) .


Probably best to start with a micro kernel design, as you can always move parts into kernel space later (and make it a hybrid) if you have issues.

_________________
CuriOS: A single address space GUI based operating system built upon a fairly pure Microkernel/Nanokernel. Download latest bootable x86 Disk Image: https://github.com/h5n1xp/CuriOS/blob/main/disk.img.zip
Discord:https://discord.gg/zn2vV2Su


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 2:52 pm 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
Good thought, @bloodline. I have been leaning towards a microkernel anyway, I'll go for that.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 3:05 pm 
Offline
Member
Member
User avatar

Joined: Tue Sep 15, 2020 8:07 am
Posts: 264
Location: London, UK
nexos wrote:
Good thought, @bloodline. I have been leaning towards a microkernel anyway, I'll go for that.


I’m obviously biased as I prefer the micro kernel design, but I think it’s much easier to start with a micro kernel and then add monolithic features to it than start with a monolithic kernel and try and get it to exhibit micro kernel like behaviour...

If I’m not very much mistaken, all the major modern hybrid kernels (which I think covers almost every OS other than Linux) started out as pure micro kernel designs!

_________________
CuriOS: A single address space GUI based operating system built upon a fairly pure Microkernel/Nanokernel. Download latest bootable x86 Disk Image: https://github.com/h5n1xp/CuriOS/blob/main/disk.img.zip
Discord:https://discord.gg/zn2vV2Su


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 6:26 pm 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 399
nexos wrote:
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out :) .


You could call it NexHurd :)


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Tue Dec 22, 2020 10:12 pm 
Offline
Member
Member

Joined: Wed Apr 01, 2020 4:59 pm
Posts: 73
nexos wrote:
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently.


The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Wed Dec 23, 2020 1:39 am 
Offline
Member
Member
User avatar

Joined: Tue Sep 15, 2020 8:07 am
Posts: 264
Location: London, UK
moonchild wrote:
nexos wrote:
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently.


The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.


Obviously I’m not an expert, but when I read the papers regarding poor performance of the micro kernel design vs a monolithic design, they focus on problems caused by having a single CPU. When you can have when servers and tasks can genuinely run concurrently (i.e. on an SMP Machine), these performance issues become negligible, and more than this, the design of the micro kernel lends itself to multiple threads of concurrent execution.

_________________
CuriOS: A single address space GUI based operating system built upon a fairly pure Microkernel/Nanokernel. Download latest bootable x86 Disk Image: https://github.com/h5n1xp/CuriOS/blob/main/disk.img.zip
Discord:https://discord.gg/zn2vV2Su


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Wed Dec 23, 2020 6:08 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
thewrongchristian wrote:
nexos wrote:
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently. Luckily, I plan to write the user space utilities (i.e., ls, cp, mv, rm, wc, and so on) plus a filesystem and partition managing app first, so I have time to figure this out :) .


You could call it NexHurd :)

I like it :) . It is kind of going to be like a modern GNU.
moonchild wrote:
The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes

Yeah, I've been thinking about the performance a bit, and trying to mitigate some concern. I think PCID will help some, I plan on writing a user threading system integrated with kernel threads, and several scheduler optimizations to mitigate these concerns. However, a distributed monolithic kernel sounds hard :) . I think that is a microkernel's big advantage.

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Wed Dec 23, 2020 6:22 am 
Offline
Member
Member

Joined: Tue Apr 03, 2018 2:44 am
Posts: 399
bloodline wrote:
moonchild wrote:
nexos wrote:
OK, I have a question, and no flaming here :) . For what I want, I am debating between a hybrid kernel and a microkernel. I want my kernel to scale well to large NUMA clusters, use asynchronous I/O, and support distributed systems. What would you choose for that kind of system? I also want good structure, and an interesting project, so I am leaning in the microkernel direction currently.


The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.


Obviously I’m not an expert, but when I read the papers regarding poor performance of the micro kernel design vs a monolithic design, they focus on problems caused by having a single CPU. When you can have when servers and tasks can genuinely run concurrently (i.e. on an SMP Machine), these performance issues become negligible, and more than this, the design of the micro kernel lends itself to multiple threads of concurrent execution.


Aren't performance "problems" with early micro-kernels basically down to the fact they operated synchronously? That way, each message is guaranteed to result in a context switch, and all that that implies.

Even with asynchronous messaging, the problem then becomes one of implementing compatible APIs on top that are inherently synchronous, such as POSIX. But this translation can also be a benefit, as it gives scope to provide a more flexible user thread model if the OS interface is inherently asynchronous, so making N:M threading models not only easier to implement, but also natural.

And of course, when operating in messages, it doesn't matter where the message consumer lives, so it can be a kernel thread in the case of a hybrid kernel, or a user thread in the case of a purer micro-kernel. But in either case, you can scale across multiple CPUs to achieve performance through concurrency.

Doesn't this basically describe managarm?


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Wed Dec 23, 2020 6:57 am 
Offline
Member
Member

Joined: Tue Feb 18, 2020 3:29 pm
Posts: 1071
thewrongchristian wrote:
bloodline wrote:
moonchild wrote:
The main problem with microkernels is performance. However, they don't scale any worse than traditional kernels; so I think they would work well for your purposes.


Obviously I’m not an expert, but when I read the papers regarding poor performance of the micro kernel design vs a monolithic design, they focus on problems caused by having a single CPU. When you can have when servers and tasks can genuinely run concurrently (i.e. on an SMP Machine), these performance issues become negligible, and more than this, the design of the micro kernel lends itself to multiple threads of concurrent execution.


Aren't performance "problems" with early micro-kernels basically down to the fact they operated synchronously? That way, each message is guaranteed to result in a context switch, and all that that implies.

Even with asynchronous messaging, the problem then becomes one of implementing compatible APIs on top that are inherently synchronous, such as POSIX. But this translation can also be a benefit, as it gives scope to provide a more flexible user thread model if the OS interface is inherently asynchronous, so making N:M threading models not only easier to implement, but also natural.

And of course, when operating in messages, it doesn't matter where the message consumer lives, so it can be a kernel thread in the case of a hybrid kernel, or a user thread in the case of a purer micro-kernel. But in either case, you can scale across multiple CPUs to achieve performance through concurrency.

Doesn't this basically describe managarm?

Yeah, that makes sense. I read an article a couple days ago about the only thing prevent computers with 100000+ nodes is POSIX I/O. Basically, it was saying POSIX I/O scales horribly, and I have to agree. Anyway, I think I have an architectural plan for my kernel, tell me any issues you see in it :)
So, my idea is an "everything is a message port idea". I'm going to revive the old Mach port idea, and message ports won't necessarily represent a message queue. It could represent something like a semaphore, a process control block, any resource inside the kernel is going to be a message port. A port representing one of these objects doesn't context switch, so it contains hardcoded function pointers called message handlers. The message manager takes the message ID, and uses that to find this messages messages handler, passing the message packet as a parameter. Now the object we sent the message to acts upon it, and returns, all without a context switch. Those are user mode to kernel mode messages. Then there are user mode to user mode messages, which work in the normal way. What do you say about that?

_________________
"How did you do this?"
"It's very simple — you read the protocol and write the code." - Bill Joy
Projects: NexNix | libnex | nnpkg


Top
 Profile  
 
 Post subject: Re: I keep recoding my OS!
PostPosted: Wed Dec 23, 2020 9:33 am 
Offline
Member
Member

Joined: Fri Nov 22, 2019 5:46 am
Posts: 590
You probably mean this article:
https://www.nextplatform.com/2017/09/11 ... -posix-io/
It's interesting read.

I like the general idea of Hurd, but I must admit that Mach makes me dizzy.

Greetings
Peter


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 34 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 9 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group