OSDev.org

The Place to Start for Operating System Developers
It is currently Fri Mar 29, 2024 8:16 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Open source Midori?
PostPosted: Fri Nov 20, 2015 5:46 am 
Offline
Member
Member
User avatar

Joined: Tue Oct 17, 2006 11:33 pm
Posts: 3882
Location: Eindhoven
I think some of you may have seen discussions flying around about Midori. It is hitting *very* close to home for me; it's practically the exact design I had in mind and that I was working out the last year or two.

Given that it has a lot of the same choices, I think it has a good theoretical basis to continue on. So is anybody else inspired by their proposals and if so, would there be enough interest to start up a group project for an open source Midori OS?

http://joeduffyblog.com/2015/11/19/asyn ... verything/ plus previous posts.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Sat Nov 21, 2015 6:47 am 
Offline
Member
Member
User avatar

Joined: Tue Oct 17, 2006 11:33 pm
Posts: 3882
Location: Eindhoven
... No interest at all?


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Sat Nov 21, 2015 8:49 am 
Offline
Member
Member

Joined: Sat Nov 07, 2015 3:12 pm
Posts: 145
Hello, I just read the article. I'm quite shoked actually, because I wanted to try OS development to try a task oriented kernel architecture. ( with primitives like continueWith and cancellation tokens)
IMHO, asynchronous tasks are powerful, but often misused.
In my kernel, I wanted asynchronous tasks for everything that has an undefined time of execution, or thing that could fail if we wanted immediate resolution.

For example, I wanted the sbrk function to take a token of cancellation, meaning the sbrk function could block until physical memory become available or until the application decide it has waited too much.

Unfortunately for you, im new in OS development. for now, I have only a two part kernel, the first part activates paging and loads the second part. But I'm interested.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Sun Nov 22, 2015 5:38 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9301
Location: On the balcony, where I can actually keep 1½m distance
I think time is my personal issue, rather than it being a project I would not see myself work on.

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Tue Nov 24, 2015 6:32 pm 
Offline
Member
Member

Joined: Wed Nov 10, 2010 10:55 pm
Posts: 61
I am interested in many of the ideas worked on in Midori. In particular, I'm a fan of capability-based authority and async interfaces. I think a big drawback with their design is that it cannot support legacy software - they use (compiler-enforced) software isolation, and cannot support blocking code. I'm looking into ways to get the benefits without these drawbacks.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 2:12 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

dschatz wrote:
I am interested in many of the ideas worked on in Midori. In particular, I'm a fan of capability-based authority and async interfaces. I think a big drawback with their design is that it cannot support legacy software - they use (compiler-enforced) software isolation, and cannot support blocking code. I'm looking into ways to get the benefits without these drawbacks.


When Midori's developers say "everything is non-blocking" what they really mean is that waiting for a message to arrive is the only thing that causes a thread to block; and because everything waits for messages to arrive everything is blocking. ;)


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 3:02 am 
Offline
Member
Member
User avatar

Joined: Sun Sep 19, 2010 10:05 pm
Posts: 1074
I had not realized that Midori had been "abandoned", although I had pretty much assumed it because of the lack of updates over the past few years.

One of the reasons I started my own project was because MS had abandoned their Singularity project, and I wanted to see what a "managed" OS would look like. So I started with the idea of an Object based OS, and worked backwards to figure out the quickest way to get there. My biggest goal was to be able to allow the end user to create and manipulate objects from a text-based command prompt, and then expand from there. Once the user could create objects and call methods on them, the rest has been pretty much just creating more classes to talk to more hardware, and to perform common tasks like downloading data from the Internet, and parsing file system tables.

I'm pretty happy with the way things have worked out, but there's still a ton of work to do, and I've still got a lot of ideas that I haven't even started implementing yet. But, let us know what you decide on your project.

_________________
Project: OZone
Source: GitHub
Current Task: LIB/OBJ file support
"The more they overthink the plumbing, the easier it is to stop up the drain." - Montgomery Scott


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 6:22 am 
Offline
Member
Member

Joined: Wed Nov 10, 2010 10:55 pm
Posts: 61
Brendan wrote:
When Midori's developers say "everything is non-blocking" what they really mean is that waiting for a message to arrive is the only thing that causes a thread to block; and because everything waits for messages to arrive everything is blocking. ;)


The article says
Joe Duffy wrote:
Because in Midori the only way to "wait" for something was to use the asynchronous model, and there was no hidden blocking
My interpretation of this is that you cannot "wait for messages" but rather you provide an explicit dataflow to occur when a message arrives. So suppose I want to port code that uses a synchronous read(). As a developer, I need to implement read() on top of the provided asynchronous read, but they don't provide any blocking wait. I see no way to preserve the stack until the asynchronous read is complete.

Now this has its advantages - namely, when I invoke some component, I can be reasonably certain that control will be returned to me in a timely matter. But a drawback is that it cannot support legacy, blocking code.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 7:13 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9301
Location: On the balcony, where I can actually keep 1½m distance
dschatz wrote:
But a drawback is that it cannot support legacy, blocking code.
You'd have to go to quite some lengths to get a compiler to convert all I/O calls to continuations (or run a vm under the same merit), but supporting legacy is not strictly impossible.

Then again, getting rid of some legacy can be a good thing.

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 8:02 am 
Offline
Member
Member

Joined: Wed Nov 10, 2010 10:55 pm
Posts: 61
Combuster wrote:
You'd have to go to quite some lengths to get a compiler to convert all I/O calls to continuations (or run a vm under the same merit), but supporting legacy is not strictly impossible.

Then again, getting rid of some legacy can be a good thing.

I don't think it's sufficient for a compiler to convert all I/O calls. It also needs to convert all function calls in a call stack that may block. Basically, turning blocking code into non-blocking code requires duplicating the entire call chain around the block. This is sometimes called "stack-ripping". Using a VM is feasible but makes it difficult to compose legacy code with new code (e.g. use existing libraries). In general, though, I agree, getting rid of some legacy is good, but my own philosophy is that a successful approach will be incremental, not reboot_the_world().


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 8:39 am 
Offline
Member
Member
User avatar

Joined: Tue Oct 17, 2006 11:33 pm
Posts: 3882
Location: Eindhoven
dschatz wrote:
Combuster wrote:
You'd have to go to quite some lengths to get a compiler to convert all I/O calls to continuations (or run a vm under the same merit), but supporting legacy is not strictly impossible.

Then again, getting rid of some legacy can be a good thing.

I don't think it's sufficient for a compiler to convert all I/O calls. It also needs to convert all function calls in a call stack that may block. Basically, turning blocking code into non-blocking code requires duplicating the entire call chain around the block. This is sometimes called "stack-ripping". Using a VM is feasible but makes it difficult to compose legacy code with new code (e.g. use existing libraries). In general, though, I agree, getting rid of some legacy is good, but my own philosophy is that a successful approach will be incremental, not reboot_the_world().


I've given some thought how to port GCC to such a system. GCC has blocking operations - when you start thinking about it even printf() and read() must be asynchronous - so to support it you have to be able to block.

You basically invert the common paradigm of implementing tasks on a thread pool - you implement threads as tasks that have their own stack ownership and that manually switch back to the regular stack when suspending, and have a continuation attached that switches back & continues the thread. Then you have a thread that is implemented as a sequence of tasks. You also make wrappers for all asynchronous functions that are synchronous and that link to the thread suspension mechanism - basically, make async call, attach continuation for resume, return.

You do a similar thing for new applications - you replace main() with a task entry point task_main() which takes a shared_ptr to the application as argument (or equivalent in whatever your language is). When that application object is deallocated everything is cleaned up for that application instance. That allows you to have literally zero threads in most applications.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 9:16 am 
Offline
Member
Member

Joined: Wed Nov 10, 2010 10:55 pm
Posts: 61
Candy wrote:

I've given some thought how to port GCC to such a system. GCC has blocking operations - when you start thinking about it even printf() and read() must be asynchronous - so to support it you have to be able to block.

You basically invert the common paradigm of implementing tasks on a thread pool - you implement threads as tasks that have their own stack ownership and that manually switch back to the regular stack when suspending, and have a continuation attached that switches back & continues the thread. Then you have a thread that is implemented as a sequence of tasks. You also make wrappers for all asynchronous functions that are synchronous and that link to the thread suspension mechanism - basically, make async call, attach continuation for resume, return.

You do a similar thing for new applications - you replace main() with a task entry point task_main() which takes a shared_ptr to the application as argument (or equivalent in whatever your language is). When that application object is deallocated everything is cleaned up for that application instance. That allows you to have literally zero threads in most applications.

This is what stackful coroutines are. To be explicit, this is a user-level thread with much of the associated problems therein - e.g. how does mmap()ing a file work? What happens when a coroutine page faults? Typically the kernel is unaware of the user-level threading and suspends the entire (kernel) thread despite the fact that other coroutines are available.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 10:08 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9301
Location: On the balcony, where I can actually keep 1½m distance
Quote:
mmap()
Of course, there's no point in discussing issues with synchronous system calls on a kernel that has no synchronous system calls :wink:

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 10:37 am 
Offline
Member
Member

Joined: Wed Nov 10, 2010 10:55 pm
Posts: 61
Combuster wrote:
Quote:
mmap()
Of course, there's no point in discussing issues with synchronous system calls on a kernel that has no synchronous system calls :wink:

Just to be clear, it doesn't much matter if the call to mmap() itself is synchronous or asynchronous, but the demand paging aspect of it is where the problem lies. The solution, of course, is to implement mmap() as if mlock() occured afterwards so that faults cannot happen. This comes with performance issues.


Top
 Profile  
 
 Post subject: Re: Open source Midori?
PostPosted: Wed Nov 25, 2015 11:17 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8561
Location: At his keyboard!
Hi,

dschatz wrote:
Brendan wrote:
When Midori's developers say "everything is non-blocking" what they really mean is that waiting for a message to arrive is the only thing that causes a thread to block; and because everything waits for messages to arrive everything is blocking. ;)


The article says
Joe Duffy wrote:
Because in Midori the only way to "wait" for something was to use the asynchronous model, and there was no hidden blocking
My interpretation of this is that you cannot "wait for messages" but rather you provide an explicit dataflow to occur when a message arrives. So suppose I want to port code that uses a synchronous read(). As a developer, I need to implement read() on top of the provided asynchronous read, but they don't provide any blocking wait. I see no way to preserve the stack until the asynchronous read is complete.


My interpretation is that you can "wait for messages" (and it does block), but they've obfuscated the event loop under syntactical sugar ("await") in an attempt to make the actor model seem like procedural programming.

dschatz wrote:
Now this has its advantages - namely, when I invoke some component, I can be reasonably certain that control will be returned to me in a timely matter. But a drawback is that it cannot support legacy, blocking code.


For legacy non-blocking code, I think you'd just do something like (e.g.) "fp = await Async.Open(filename);".


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: Google [Bot] and 30 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group