OSDev.org

The Place to Start for Operating System Developers
It is currently Wed Aug 22, 2018 2:17 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 9 posts ] 
Author Message
 Post subject: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 1:41 am 
Offline

Joined: Wed Feb 22, 2017 4:57 pm
Posts: 8
Hello,

I have had this idea brewing for the past year or so. I want to write an operating system which is powerful and elegant at the same time. I want to call it nOS, short for "next Operating System". Call it a throwback to the NeXTSTEP systems, as I was also inspired by it in parts. The reason I am creating this thread is not to get people to create my operating system for me, but rather to get my design peer reviewed, in case I am missing something or other, or there is a flaw that I don't see but others might.

Philosophy
nOS should be designed with elegance in mind. While performance is also a concern, the elegance of the system should not be cut back. This elegance not only applies to the general user experience and the graphical interface, but also the programming environment of nOS; it should be fun to program an application for the nOS environment. The nOS programming environment should provide as much support as possible for programmers with many native libraries (sound, graphics, networking, ...) provided with a consistent, namespaced and standard interface.
If a compromise must be made from elegance for performance reasons, the user should be given both options.

Kernel Design
The nOS kernel, codename Spark, will be based on the microkernel architecture. Services should lay on the userspace (unless the required functionality is crucial to system performance), and can provide anything; including drivers, timers, watchdogs, regular daemons, etc.
The nOS programming environment will provide a standard set of services for accessing the hardware. Additionally, new services can be created and added just like the native ones, to access new hardware.
The kernel should, ideally, consist of system bring-up code, a scheduler, and memory management. Additional parts can be added based on CPU architecture and hardware.
The kernel (?) should also provide a system-wide bus which services and programs use to communicate to each other. The bus should be as high performance as possible.
(Just an idea: should the scheduler try to move a service that has been called by the running task at the top of the task list right after the current running tasks? This way service request latency could be minimized.)

nOS Programming Language and Environment
The nOS programming language will be nLang (currently not public, and syntax not finalized), which is a managed programming language. Here is a hello world program:
Code:
[[$]] hello_world.
$<[[io]]::print.

;; Top level instructions are executed directly; no need for a main. The argument handling is different from Unix-likes, more on that in detail later.
;; Functions take a single argument. Functions that take multiple arguments will use tuples.
$::print"Hello World!".

For a more complex example, here's a GUI window (pseudo-code):
Code:
[[$]] my_window.
$::[[containers]]::M. ;; map factory instance. overrides operator `[]` to generate a map from and to (types are objects). See below.
$<[[ui]]::[Window Text Button Event].
$<[[io]]::print.

;; note: strings are first class objects.
;; a little more about the language: the language supports complete type reflection, runtime variables. However dynamic methods and dynamic members are not supported.

window := $::Window::new!. ;; Function`!` is shorthand for func {} (calling function with empty tuple). Optimized by the compiler.
window::child {$::Text::new {
  content = "Hello GUI!"
  anchor = window::anchor::top_center
  position = [0 20] ;; X Y, anchored from the top center edge of the window. [] is array; type of the position attribute is i32[2]
}}.
window::child {$::Button::new {
  text = "Close Window"
  events = M[string][%{Event}->bool] { ;; %{Event}->bool is function taking an Event and returning a bool. the boolean means whether to propagate the event to objects the button is contained in.
    "click" %({e: Event} bool -> ;; The compiler will capture all variables that the function uses. No eval, so static analysis can determine. VM ensures window stays alive while captured.
      window::close!.
      false) ;; Last evaluated expression is the return value.
  }
  anchor = window::anchor::bottom_center
  position = [0 -20]
}}.

;; The window will configure its own buttons, and signals from the windowing system.
window::wait_for_events!.


nOS Standard Library
The nOS Standard Library is not POSIX compatible. A compatibility shim will be provided, and C programs should still work, but the system does not aim to be POSIX compatible.
The standard library should contain as many native faculties as necessary for the full programming potential of the entire system. A list of what should be provided (inconclusive):
Containers (list, map, vector, ...), string manipulation, I/O, 2D canvas drawing, 3D context drawing (OpenGL), sound libraries, windowing system, networking, type reflection, service access, bus access, IPC, regular expressions, authentication system, ...

The Windowing System
The windowing system is codenamed Crystal. The windows and UI elements should use the frosted glass effect when appropiate, to give depth. Shadows should also be used. Font kerning and anti-alias is allowed, to some degree. The menus should be quickly accessible and well organized. The window system should be able to switch between tiled and floating windows at will. 3D can be used to denote depth, and subtle animations should be used where appropiate.
The boot sequence should be seamless. After the boot sequence ends, the log-in screen should fade in, and ideally after the user logs in, the windowing system should slide in.

The Standard Library API
This was my main gripe with a lot of systems, so this is something that I want to fix with my operating system's standard library. Many operating systems' APIs (be it Win32, POSIX, etc.) suffer from a lot of backwards compatibility kludges, which are kept because the operating system wants to keep compatible with old programs while new versions are still released. My idea is to specify which version of the operating system is wanted, through some means.
Here's an example: say my operating system's API version 1 has a function which is deemed buggy, and the fix would require code change. The next version of nOS API (version 2) would fix this function, and new applications written for nOS would specify they want API 2. API 1's buggy function would have a workaround that would not require code change, while API 2 would have a proper fix. I believe this is already used by Google's Android, which has a different API version for each minor and major release of its OS.

This was an hour of typing, hopefully it is mostly coherent. I will add more as I collect my thoughts.


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 3:27 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8453
Location: At his keyboard!
Hi,

m712 wrote:
Kernel Design
The nOS kernel, codename Spark, will be based on the microkernel architecture. Services should lay on the userspace (unless the required functionality is crucial to system performance), and can provide anything; including drivers, timers, watchdogs, regular daemons, etc.


How do processes (drivers, services, applications) communicate with each other? For micro-kernels this is very important.

m712 wrote:
The kernel should, ideally, consist of system bring-up code, a scheduler, and memory management. Additional parts can be added based on CPU architecture and hardware.


What kind of "bring up code"? Note that typically a micro-kernel has no video drivers (to display error messages, etc), no storage device drivers and no file system (to start other drivers, etc) built-in, and typically relies on boot code to provide all of that, and typically that boot code needs to do some memory management before kernel is started; so normally the kernel's own initialisation involves initialising the scheduler and doing "memory management hand-off" and not much else.

Typically the kernel also handles timers, partly because the scheduler needs them (for measuring time slices, etc) and so that things like "nano_sleep_until()" can be implemented; and also includes some way for processes in user-space to communicate with each other (often including some kind of "wait_for_message()"; where both of these things (sleeping and communication) causes tasks/threads to block and unblock and ends up closely tied to the scheduler.

m712 wrote:
The Standard Library API
This was my main gripe with a lot of systems, so this is something that I want to fix with my operating system's standard library. Many operating systems' APIs (be it Win32, POSIX, etc.) suffer from a lot of backwards compatibility kludges, which are kept because the operating system wants to keep compatible with old programs while new versions are still released. My idea is to specify which version of the operating system is wanted, through some means.


Operating systems don't have standard libraries - only programming languages have them. Your kernel will provide some kind of API, and any language (with whatever standard library it comes with, if any) will use that kernel API; and your services and drivers will provide some kind of interface (messaging protocol?) and any language (with whatever standard library it comes with, if any) will use those interfaces too.

m712 wrote:
Here's an example: say my operating system's API version 1 has a function which is deemed buggy, and the fix would require code change.


For micro-kernels; the operating system doesn't have one API. It has one "kernel API" (which doesn't do much more than scheduling, communication and memory management only because it is a micro-kernel) ; and then you'd have lots of different kinds of interfaces/protocols for all the different types of services and drivers (e.g. "stream IO protocol" and "random access storage IO protocol" and "GUI protocol" and "printer driver protocol" and ..).

If (e.g.) the user replaces an older sound card driver with a newer one; they'd terminate the old driver (causing everything that was using "sound driver protocol version 1" to communicate with the old driver to disconnect from it) and then start the new driver (hopefully causing everything that wants to use it to try to re-establish a connection), and after some negotiations to see which versions of which protocols they both support they'd start using "sound driver protocol version 2". You'd also need some overlap to ensure that everything isn't constantly broken (e.g. a sound driver might only support one version of the sound protocol and an application might only support one version of the sound protocol, but you might have a "sound manager" service in the middle that supports 4 versions of the protocol).

Of course you could also have 10 applications all written at different times that happen to be using 10 different versions of the standard library; but that has everything to do with the language and nothing to do with the OS itself.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 3:48 am 
Offline

Joined: Wed Feb 22, 2017 4:57 pm
Posts: 8
Brendan wrote:
How do processes (drivers, services, applications) communicate with each other? For micro-kernels this is very important.

I thought of providing a system-wide bus that processes can use to communicate with each other.
Brendan wrote:
What kind of "bring up code"? Note that typically a micro-kernel has no video drivers (to display error messages, etc), no storage device drivers and no file system (to start other drivers, etc) built-in, and typically relies on boot code to provide all of that, and typically that boot code needs to do some memory management before kernel is started; so normally the kernel's own initialisation involves initialising the scheduler and doing "memory management hand-off" and not much else.

I meant bring up code, as in setting up the CPU and preparing for the userspace, aka abstracting away the hardware phase.

[code]Typically the kernel also handles timers, partly because the scheduler needs them (for measuring time slices, etc) and so that things like "nano_sleep_until()" can be implemented; and also includes some way for processes in user-space to communicate with each other (often including some kind of "wait_for_message()"; where both of these things (sleeping and communication) causes tasks/threads to block and unblock and ends up closely tied to the scheduler.[/quote]
The timer service could be directly provided by the kernel, I guess that is fine. The bus should be used for IPC.

Brendan wrote:
Operating systems don't have standard libraries - only programming languages have them. Your kernel will provide some kind of API, and any language (with whatever standard library it comes with, if any) will use that kernel API; and your services and drivers will provide some kind of interface (messaging protocol?) and any language (with whatever standard library it comes with, if any) will use those interfaces too.

Then I should call it an "operating environment". I expect everything to be one coherent whole (still individually separable, but comes in one package).

Brendan wrote:
For micro-kernels; the operating system doesn't have one API. It has one "kernel API" (which doesn't do much more than scheduling, communication and memory management only because it is a micro-kernel) ; and then you'd have lots of different kinds of interfaces/protocols for all the different types of services and drivers (e.g. "stream IO protocol" and "random access storage IO protocol" and "GUI protocol" and "printer driver protocol" and ..).

See above.

Really appreciate the evaluation.

Now here's an idea that's not popular: in my operating system IPC will use a standard data structure-based sharing system. This way IPC is not limited to text only. This can be extended to be used in the shell, where you can finally have standardized arguments with a structure passed to programs. People usually mention PowerShell when I say this but I find that to be very kludgy.
Also, after studying some of the ideas of TempleOS (:shock:) a bit, I thought of adding 1) a similar document format that extends upon not ASCII but UTF-8, and 2) JIT compiler/interpreter in the shell. For 1), Unicode already has Private Use Areas specifically designed for this, so implementing a markup language there shouldn't be much of a problem, I guess. 2) is a little bit tricky, though. Respect to Terry.


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 4:58 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8453
Location: At his keyboard!
Hi,

m712 wrote:
Brendan wrote:
How do processes (drivers, services, applications) communicate with each other? For micro-kernels this is very important.

I thought of providing a system-wide bus that processes can use to communicate with each other.


What is "a bus"? Specifically; how is the data transferred, and how does it interact with the scheduler?

For a random example; does sending data to the bus cause every other thread in the entire OS to be woken up (if they were waiting for data) and given their own separate copy of the sender's data (so that if you send a 2345-byte piece of data to the bus and there are 6789 threads running, it costs 16 MiB of RAM for all the copies and three hours for all the task switching)?

m712 wrote:
Brendan wrote:
For micro-kernels; the operating system doesn't have one API. It has one "kernel API" (which doesn't do much more than scheduling, communication and memory management only because it is a micro-kernel) ; and then you'd have lots of different kinds of interfaces/protocols for all the different types of services and drivers (e.g. "stream IO protocol" and "random access storage IO protocol" and "GUI protocol" and "printer driver protocol" and ..).

See above.

Really appreciate the evaluation.

Now here's an idea that's not popular: in my operating system IPC will use a standard data structure-based sharing system.


That would imply that the IPC is some form of messages or datagrams (structures as the smallest unit of communication) rather than streams/pipes (bytes as the smallest unit of communication).

m712 wrote:
This way IPC is not limited to text only. This can be extended to be used in the shell, where you can finally have standardized arguments with a structure passed to programs. People usually mention PowerShell when I say this but I find that to be very kludgy.
Also, after studying some of the ideas of TempleOS (:shock:) a bit, I thought of adding 1) a similar document format that extends upon not ASCII but UTF-8, and 2) JIT compiler/interpreter in the shell. For 1), Unicode already has Private Use Areas specifically designed for this, so implementing a markup language there shouldn't be much of a problem, I guess. 2) is a little bit tricky, though. Respect to Terry.


If the JIT compiler/interpreter is in a shell, does that mean the kernel has to support native executables (so that it can execute the shell); and if the kernel has to support native executables then why can't other executables also use native code? Are you saying that all modern applications (that are started from the GUI and don't use a shell at all) need to be native code?


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 5:08 am 
Offline

Joined: Wed Feb 22, 2017 4:57 pm
Posts: 8
Brendan wrote:
What is "a bus"? Specifically; how is the data transferred, and how does it interact with the scheduler?

I would have to study how IPC buses work more to give you a definite answer, but my initial thought was a message passing service (not THAT kind of service) that will pass messages from a source (PID? org.freedesktop.MyApplication?) to a target, using the data format specified by the OS. Not the horrifying situation you described, though :D

Brendan wrote:
That would imply that the IPC is some form of messages or datagrams (structures as the smallest unit of communication) rather than streams/pipes (bytes as the smallest unit of communication).

Yes, that is what I was going for. Is there any caveats with this?

Brendan wrote:
If the JIT compiler/interpreter is in a shell, does that mean the kernel has to support native executables (so that it can execute the shell); and if the kernel has to support native executables then why can't other executables also use native code? Are you saying that all modern applications (that are started from the GUI and don't use a shell at all) need to be native code?

The kernel doesn't have to support native executables. The JIT can also compile to a bytecode target which the kernel VM (?) can execute. But then, how would C programs run? Obviously ELF binaries need to be supported if that is a requirement, and I don't really want to write my own web browser! 8-[


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 6:06 am 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8453
Location: At his keyboard!
Hi,

m712 wrote:
Brendan wrote:
What is "a bus"? Specifically; how is the data transferred, and how does it interact with the scheduler?

I would have to study how IPC buses work more to give you a definite answer, but my initial thought was a message passing service (not THAT kind of service) that will pass messages from a source (PID? org.freedesktop.MyApplication?) to a target, using the data format specified by the OS. Not the horrifying situation you described, though :D


If it passes messages from source to target (and doesn't broadcast messages from source to lots of targets), then it's not really a bus.

m712 wrote:
Brendan wrote:
That would imply that the IPC is some form of messages or datagrams (structures as the smallest unit of communication) rather than streams/pipes (bytes as the smallest unit of communication).

Yes, that is what I was going for. Is there any caveats with this?


It's fine (and its my preference, so that's nice).

Note that the goal here is to refine the description into something concrete. So far we've got down to "IPC is some sort of messaging". The next step is to define the interactions with the scheduler. For this there are 2 main categories - "synchronous messaging" (sender must block until it receives a reply) and "asynchronous messaging" (sender may keep running without blocking/waiting for reply).

m712 wrote:
Brendan wrote:
If the JIT compiler/interpreter is in a shell, does that mean the kernel has to support native executables (so that it can execute the shell); and if the kernel has to support native executables then why can't other executables also use native code? Are you saying that all modern applications (that are started from the GUI and don't use a shell at all) need to be native code?

The kernel doesn't have to support native executables. The JIT can also compile to a bytecode target which the kernel VM (?) can execute. But then, how would C programs run? Obviously ELF binaries need to be supported if that is a requirement, and I don't really want to write my own web browser! 8-[


Anything can be compiled to byte-code (including C - e.g. clang compiles C into LLVM byte-code); the problem is that it'd be hard to compile some languages (C, C++, ...) into managed byte-code rather than "unmanaged byte-code" (but that doesn't mean you can't use " primarily managed byte-code, but with optional support for unsafe/unmanaged pieces that can be disabled").

The VM could be implemented as a kind of "executable loader" in user-space or implemented in the kernel (although that's not very "micro-kernel-ish"); and in those cases kernel wouldn't need to support native code (except for the VM itself, which would be a special/trusted thing wherever its implemented), and you could write the shell in your managed language.

You determine the operating system's requirements, so you determine if supporting ELF binaries is a requirement. Note that when you're starting an executable you can determine the type of file and start whatever "executable loader" makes sense for that type of file; which could allow anyone to add support for any kind of executable file (including native ELF executables, and any kind of byte-code VM, and scripting languages, and ...) whenever they like.

It'll probably take many years (writing kernel, compiler, VM, drivers, shell, ...) before you have to worry about things like web browsers.


Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 10:49 am 
Offline
Member
Member
User avatar

Joined: Fri Feb 17, 2017 4:01 pm
Posts: 308
Location: Ukraine, Bachmut
Quote:
nOS Programming Language and Environment
The nOS programming language will be nLang (currently not public, and syntax not finalized), which is a managed programming language. Here is a hello world program:

oh dear. what the processor is gonna be in all this napoleonic plan? I suppose you design the CPU architecture too? :mrgreen: What's the codename? nCube? iYayay? AquaX? :lol:
honestly, it is not freaking serious.

Recently I managed to write a tiny first form of an OS loader on 64 bit ARM and run it on the real board, not those stupid virtual sh1t. not that I think i'm cool, not at all. there are people over here that did sooo much, that it really inspires. I wanted to say, it's too complex. Trying to do everything is unfeasible. So far it looks you devised only those numerous codenames. It's not easy, I know, but it's not the hardest thing out here. Just lower your plan a little. Otherwise it just will burst as a bubble and you drop it.

_________________
future big goal: ANT - NT-like OS for mips, arm and x86.
current smaller goal: efify - UEFI for a couple of boards (mips and arm).


Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 12:21 pm 
Offline

Joined: Wed Feb 22, 2017 4:57 pm
Posts: 8
Brendan wrote:
If it passes messages from source to target (and doesn't broadcast messages from source to lots of targets), then it's not really a bus.

Do I just call it IPC then? I intend to use the same system with driver services.
From what I understand, buses broadcast instead of delivering, right?

Brendan wrote:
It's fine (and its my preference, so that's nice).
Note that the goal here is to refine the description into something concrete. So far we've got down to "IPC is some sort of messaging". The next step is to define the interactions with the scheduler. For this there are 2 main categories - "synchronous messaging" (sender must block until it receives a reply) and "asynchronous messaging" (sender may keep running without blocking/waiting for reply).

I think that since I don't want to constrain myself to the limits of Weenix, I would like to wait asynchronously using anonymous handler functions that handle messages.

Brendan wrote:
Anything can be compiled to byte-code (including C - e.g. clang compiles C into LLVM byte-code); the problem is that it'd be hard to compile some languages (C, C++, ...) into managed byte-code rather than "unmanaged byte-code" (but that doesn't mean you can't use " primarily managed byte-code, but with optional support for unsafe/unmanaged pieces that can be disabled").
The VM could be implemented as a kind of "executable loader" in user-space or implemented in the kernel (although that's not very "micro-kernel-ish"); and in those cases kernel wouldn't need to support native code (except for the VM itself, which would be a special/trusted thing wherever its implemented), and you could write the shell in your managed language.

Maybe the scheduler can execute bytecode from the processes? The VM could be integrated into the scheduler. I would like to write as much as possible in nLang. Then comes the question, how do I set up the managed environment? Using assembly would mean replicating that code for every architecture, but otherwise I would need to write a special non-managed mode (like -ffreestanding). What comes first, chicken or the egg?

Brendan wrote:
You determine the operating system's requirements, so you determine if supporting ELF binaries is a requirement. Note that when you're starting an executable you can determine the type of file and start whatever "executable loader" makes sense for that type of file; which could allow anyone to add support for any kind of executable file (including native ELF executables, and any kind of byte-code VM, and scripting languages, and ...) whenever they like.

True. I could run the code natively or via VM based on its type.

[quote=Brendan]It'll probably take many years (writing kernel, compiler, VM, drivers, shell, ...) before you have to worry about things like web browsers[/quote]
True dat. ](*,)


Last edited by m712 on Tue Jun 05, 2018 12:26 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: My operating system design (open to suggestions)
PostPosted: Tue Jun 05, 2018 12:25 pm 
Offline

Joined: Wed Feb 22, 2017 4:57 pm
Posts: 8
zaval wrote:
oh dear. what the processor is gonna be in all this napoleonic plan? I suppose you design the CPU architecture too? :mrgreen: What's the codename? nCube? iYayay? AquaX? :lol:
honestly, it is not freaking serious.

Hahaha! I don't plan to develop my own CPU. This will only be an operating environment.

zaval wrote:
Recently I managed to write a tiny first form of an OS loader on 64 bit ARM and run it on the real board, not those stupid virtual sh1t. not that I think i'm cool, not at all. there are people over here that did sooo much, that it really inspires. I wanted to say, it's too complex. Trying to do everything is unfeasible. So far it looks you devised only those numerous codenames. It's not easy, I know, but it's not the hardest thing out here. Just lower your plan a little. Otherwise it just will burst as a bubble and you drop it.

I made my own unfinished C kernel(s) before, so I have a taste of how hard it is. I wanted to design the system beforehand for it to not become a mess. I will try to not plan too far ahead, though. :D


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group