OSDev.org

what we have is
- files that are here *and only here* (alt=nil)
- files that are here but that come from a 'permanent' location (alt=http:// ... , alt=cdrom:MyFullBackup,23Aug2004:/home/file)
- files that are no longer here (e.g. content removed) but that can be retrieved provided that the user insert the proper CD / connects to the correct machine
- files that are no longer here, but which can be restored by mixing a 'permanent file retrieval' and applying a patch atop of it
- files that are no longer here because we deleted them (and i meant *delete*, like in rm * -r -f), so we don't need to remember whereelse they are since we requested them to be gone.

A DB of burn'd CDs could help to retrieve a file that you suspect lost, but this feature has little to do with the thing that interrest us here.

The talk about OS devin' on a 8086 without mouse, etc has been moved to OSdevin' on a dinosaur thread.

Keep in mind:

I'll update this when my next refinement comes out.

• Works in layers (System, Subsystem and "Userland")
• Devices are addressed by and "@" symbol and are treated (by default) as hidden files that can only be seen when addressed directly. Devices /can/ be treated as files like in Unix/Linux.
• Support for the EXT filesystem and vFAT.
• Portabilty would be possible but not planned (by me)

Notes on "layers" design: The system is the kernel and boot time linked drivers. The subsystem is operating system specific (it contains things like run time linked drivers) and the standard c libary and such. The subsystem provides an API which (some of) link into system calls and otherstuff (tm). I've heard/read that system calls can be expensive; the middle layer would increase this price for some calls. To minamize the loss, pointers to functions will most likely make up most of the sub system in the area of filtering.

This is my most basic design as of yet and _will_ be updated.

Wow... Just read through this entire thread. There are some good ideas in here. And some ideas that have great potential if implemented correctly. I like the idea of having an object oriented operating system. I have been working on a design for about four months. I have not been able to any coding yet(mostly because I have to go to class). My ideas are still formulating in my mind, as I refine and re-refine them until they become more crisp.

I don't yet have a name. Right now the OS is called Project MBTY(Mine's Better Than Yours) which is a private joke between me and the rest of my design crew.

I will try to get a rough outline of everything drawn up by tom. and post it. I just wanted to say, that this thread has really given me some things to consider when trying to come up with a good system design.

--Alex

I'm planning on doing a three layered design:
1.Kernel Layer - This consists only of the core kernel and runs at Ring 0.
2.Device Drivers - These run at Ring 1 or 2 for security and stability. Some are seperate processes, others are libary-like. Depending on privileges, they can be accessed by the user and always by the kernel.
3.Userland - This is where user programs run as processes and library-likes. They run at Ring 3.

Call gates will probably be used to communicate between layers. The executable model is what's really important here. An executable is loaded into memory so it can be used as both a process or library depending on context. This is governed by two Boolean variables. One is a true/false for whether or not the object is a process with its own scheduler time, address space and other info. The other is a variable governing how many copies of global variables are kept. If it is true then multiple copies will be kept, one for each process that loads and calls the object, including itself if the first Boolean is true. For a false, only one copy of the global variables is kept and modified by everyone. These variables can be specified to the function to load a new library-object or make a new process, which will probably be the same function.

If a process calls another object it does so on its own schedule time. If the called object keeps multiples of its globals, the set associated with this calling process is loaded and "linked" into the address space. A loaded library-object is loaded into its caller's address space.

The device drivers function like this with the kernel as the main calling process. When they recieve data via interrupt or DMA transfer or whatnot, they'll store it internally, then wait until either their scheduler time comes or something calls them. When one of those happens they give out the data as programmed. Since they run at Ring 2 they either are paged into others' address spaces, or have their own. They must call the kernel for memory and other primitives, such as object-calling and getting interrupt handlers registered. Also, they can't make certain calls and are thus more secure and stable then Ring 0 drivers. They can be loaded and unloaded like any other process/library.

I've also developed some ideas about the GUI. The OS level graphical functions will simply be based on surfaces. The screen will be the based surface owned by the kernel, but which can be given to a program. If I use an init-like program it will get screen-surface control when it runs. Whoever owns a surface can split it into sub-surfaces and delegate control of them between processes/libraries. Whoever owns a surface controls input data flow whenever control is on that surface. Each surface will have a small border around it that will change color to show that that surface is highlighted and recieving input from keyboard, mouse, etc. It can do what it likes with that input, including route it to somebody else.

Switching input-recieving surfaces would be done by pressing Ctrl and an arrow key indicating the direction to move. Input focus would be switched to the next surface found in that direction within the current owning surface. Shift, Ctrl and Up/Down would move depth levels. What I envision is a system where input focus is split for different devices, so that one could operate an IDE with their mouse and keyboard, press a key combo, and have their keyboard switched to a console sub-surface at the bottom of the screen where they could type "make" or open another program or something like that.

The default input system will be a normal console running on the main screen surface.

Drawing under this system is incredibly easy. A surface's content will simply be a 24-bit RGB framebuffer. A surface's controller is responsible for writing to the surface frame-buffer, which is then sent to be updated onto the screen with some sort of blit-call.
Since the controller of a surface can do whatever they like with the contents, programmers could code their own GUI systems in the surfaces they get.
The filesystem will be Reiser4, with a special security attribute. The system will always look for a special sub-file to every file (except sub-files) when opening the file. If it finds this file and it is executable, it will be run with current user's privileges. If it completes and returns a success code, then and ONLY then will the main file be opened. This sub-file will be an extra supported feature and not a required one.
Block Devices will be mounted as file/directories. That is, there will be a /dev tree where currently open or usable devices will reside. A device that doesn't contain a filesystem will be a device-file like in normal *nix, with the exception that its attributes can be queried as sub-files. When a filesystem device is mounted, it will appear in /dev as a file (that the raw, unmounted data content can be read from, READ ONLY when mounted) and as a directory containing the filesystem itself. In effect, devices will be device files and mount points at the same time.
There will be another type of device, the streaming device. This takes the form of a process that calls the driver or is the driver. Its difference from a block device is that instead of being accessible as a file, one accesses it through its process. This process recieves a callback function from each accessing process/library. Whenever there is data from a Streaming Device, it calls these functions passing the data as parameters.

I've finally designed my OS the way I want it, and I'm fitting in the advice I've had from Brendan and Tim over the past year or so. I won't actually begin coding until I've typset myself a book on the kernel API, and worked on prototypes of the system calls inside VMware.

The file system I won't bother to design myself, and I'll be using the Linux Ext3 file system.

Mine is a microkernel, with just the scheduler and ISRs in the micro part of the kernel, and everything else is a dynamically loadable kernel server module. This includes the memory manager, interprocess communications, device drivers, networking, graphics sub-system and everything else.

I'm using GRUB. There will be no boot loading code whatsoever.

The kernel is 64-bit clean (I've decided to make it run on 32-bit P4 and Celeron and AMD Athlon XP systems as well), although it's targeted at Athlon 64 systems mainly.

It will feature a clean, small and consistent kernel API and internally uses message passing as its native IPC mechanism. Signals and brain-dead parts of POSIX are gone. Kernel threads are implemented like in Linux. The kernel makes no distinction between processes and threads; it just so happens some of these processes share resources.

I use a container abstraction for processes, which messages get pumped in to. Inside the container there is a program to follow, and zero or more threads.

The system, above the microkernel and server kernel processes, has a series of layers to abstract the system which is where program environments live. In theory I could build Win32, .NET, Linux, BSD, etc. environments, but I won't do that and will leave it to others if they'd like that.

Initially the system will have consoles that work using the system video or over a comm link, such as ethernet, USB or serial port.

The memory manager is as simple as possible and just allocates 4KB pages for now. It is multi-user and multiprogrammed, and the number of processes is limited by the system resources only. It favours interactive processes, although it can be tuned to favour CPU-bound processes slightly more for scientific computing applications, data acquisition, etc.

I haven't yet thought of a replacement for ioctl, and maybe my design skills aren't up to the job of a terrific replacement, but I'll give it a shot. I expect to re-write this part many times, and I'll probably throw away a few first, rough versions.

A feature of this system will be the in-kernel Lisp interpreter module, and the kernel will be extensible with Lisp.

That's all for now...

Oh yes, I forgot to mention SMP. There won't be any support for different kinds of processors, only SMP (for simplicity, I hope anyway...) It will only run on a uniprocessor system first because I won't have access to a dual or quad processor system at all, probably for years to come. But SMP will be in mind right from the beginning as I design the kernel API, system calls, etc.

I don't know what to do about NUMA or anything. It's too abstract for me to understand at this point. I need a system with NUMA so it becomes concrete and easier for me to grasp. I might bear in mind high-speed network interconnects as I design the system, anticipating 10Gb+ links between nodes for a cluster-based system design as well.

I have some designs but they are kind of rough and I am still revising them alot as I learn more about how systems work. I'm pretty sure that a hybrid (micro + mono kernel) approach is the way to go, with the MM and other often accessed things being mapped into all (including drivers) address spaces, hopefully speeding the whole thing up. However I am wondering about requiring drivers to provide basic primitives that will allow exokernel - like processses to exist.

I would also like to have some kind of event - based system, where processes can be given an event signal when they recieve input, say, through the GUI module, or when a new client connects to a socket. I was wondering if I could have the scheduler call an event when that processes time came round and then have threading done through the event manager, but this may be a little complicated.

I am definetely going to use some form of shared memory for IPC, in the form of 'communications channels' where a process can request to have a part of its memory mapped into that of another process. It could then recieve event signalls telling it that an IPC connection has been established, and the process could specify wether it wanted to recieve further events when a message was recieved (this would be slower but easier to implement). The advantage of this system other than that of it being faster due to no context switching, is that IPC messages can be sent to more than one process at a time. For example a network driver could give access to a port (Each socket could have an individual channel) to a server and a logging service at the same time.

System calls will go through one of four (for each priv level) call gates, that will check out witha trusted computing base (for security) if need be, and then call the relavent task gate or call the relavent kernel function. (mapped into all address spaces)

I won't use segmentation although I'd like to try andd see how many advantages it brings sometime, I heard OS/2 uses it to the full, have to reasearch that...

lol, I've rambled on long enough! Do you think this might possibly work? If not I know all your other designs are better anyway but hey I try!

Hi!

I have been reading this board for a long time, now I decided it's time to participate!

My os design ideas have evolved over several years, now i've come to a design that is fully based on portable virtual machine bytecode, no isolated address spaces needed any more, a relational object filesystem for storage and full network transparency for every bit of the system (so every service can be shared or consumed over the net). That's a _very_ short description, let's call it the FutureOS for now.

Well, then I realized that this design is much too extraordinary to start OS coding with it ::), and so i've put together a 'more traditional' OS design for the beginning. Maybe, in the future I could evolve this design in the direction of the FutureOS.

My current OS design is a hybrid kernel, based on objects, interfaces and servers. Any service or resource is exposed via objects, which implement interfaces. An interface describes which operations are permitted on an object, and interfaces have unique types. An object can implement multiple interfaces of different types. Finally, the objects are hosted in servers, which can be kernel level servers (kernel modules) or user level servers (processes of its own). There are also in-process-servers that behave much like a dynamic library, but with the same object-based interface.

Operations on objects are issued with a lightweight messaging system, even for kernel-to-kernel calls. It should always be the same mechanism, independent from whether it is implemented in the kernel or in a user-level server. The point is that one could exchange a service implemented in the kernel (which is probably faster to use) with a user-space version (for more security and reliability) without needing to refit any piece of software that uses this service.

So you could set up the system in a completely monolythic fashion (using in-kernel versions of all device drivers, file systems, network services and the like), and on another machine in an almost pure microkernel style (placing everything in user-space processes).

Pretty tired for now...I'll continue tomorrow!

cheers
Joe

Continued:

A few words about memory management:
For organization we have memory objects, which consist of a number of pages. Every page can either be unmapped (triggering a fault on access), or mapped to a specific page in another memory object. A memory object implements the memory-mappable-interface, which allows for these operations. At the bottom we have a special object
that represents all the physical memory. It doesn't allow to map any pages into this space, but you can map the physical pages into every other memory object. On the top, we have another set of special objects, that represent the processes' address spaces. Whenever a page gets mapped there, the memory manager adjusts the page tables respectively so that it can be accessed from that address space. Finally, any other object that implements the memory-mappable interface can act as a source for page mappings, so any file object or device object that implements the memory-mappable interface can be mapped naturally, its behaviour is completely up to the implementation. We have a quite simple, fully extendable and flexible system.

Process construction:
There is no default method for starting a process from an executable file or such, like it is done under linux or windows. A process is constructed. That means that a kernel module (or another process) creates objects for the new process itself, the address space, one or more threads and the object references that should be available from within that process. Then it maps executable code into that address space, sets the threads to their entry points, and lets the process run. Thus, the system is independant of any specific executable format, any can be used, as long as there is a suitable constructor for it. Also, different constructors can support different application models, and different numbers of entry points. The executable format i'll start with will probably be elf shared libraries.

More on objects:
From within the kernel, an object can be accessed from anywhere just with its linear address. Userspace processes however need a capability to an interface in order to send messages to the object. A capability equals to one interface, to use multiple interfaces on one object, you need multiple capabilies.

Objects are arranged in a hierarchical namespace,
where directories are objects too. Processes are given all the capabilities they need at construction time. A privileged process could own a capability to the root directory,
thus being able to lookup and use any subdirectory and objects in there (there are acls too, of course). To sandbox a process, one could give it a capability to some private subdirectory, thus preventing it from even seeing any other objects that it shouldn't deal with.

Finally, directory objects implement the directory interface, that could also be implemented by other services, like file systems or other naming facilities (over network or the like). An object does not neccessarily need an object type, however every interface has a unique type. Every object implements the generic 'object'-interface, that helps discovering information about the object, like, which other intefaces it provides. A bit like java reflection here.

I'm developing in c++, the system should be simple yet flexible. I guess this principles should provide enough flexibility. Let's see how far I'll get! Comments and critics
welcome!

cheers Joe

I did some more thinking on my OS design that I want to share with you. ;D And it seems to push me even further into my ideas of the future os!

configuration space:
There should be one standardized way of how apps/drivers/services retrieve their configuration information. This is implemented through objects again - that reside, along with all the others, in the object namespace. This configuration space covers things like services that need to be loaded and started at boot time, but also descriptions of objects that should be generated (at startup, but possibly also at other events). When you want to have a GUI from the beginning on, you place configuration elements that say we need a display and a graphical login box - in that order. The system then tries to generate these, looking at the config if there are servers available that provide such objects, loads them and instructs them to do so.

Until now, it might sound a bit like the W*ndows registry (or at least, what it was originally intended for: look at the definition of services, devices or subsystems in the registry). The main difference here is however, that my configuration objects typically should follow schemas (like xml schemas), that are documented, and the whole system is more open for the administrator. It should become the most powerful (and lowest level) tool for the admin to customize the system.

Furthermore, configuration objects don't neccessarily need to be stored on disk and parsed at system or application startup time. They can also be composed by the user, or by another program, whenever he/she/it needs a new, specific object to appear.

I have also refined my thoughts on processes and code modules. The term 'process' should not equal one running program. It should rather be named 'domain'. A domain represents a context of security - it comprises threads, capabilities and a virtual address space, as well as code modules mapped into this address space. The difference from the 'process'-notion is that one domain could actually contain multiple code modules, where at another place in the os, these code modules are placed in seperated domains. This lets the deployer decide whether he/she wants to combine a set of modules when they communicate a lot (allowing for faster communication), or seperating them for more fault tolerance. Again, this should be possible whithout changing the module's implementation.

Hehe, i should really sleep more.... ::)

cheers Joe

Well, the first bit seems interesting to me, too. Using a safe language would eliminate some problems that are typical for C/C++ like languages. Having a byte code would allow to verify the correctness of the code (as in that it is a valid program written in that language) and would eliminate the need for several address spaces then.

reactions to "virtual machines", "bytecodes", etc. within an OS internals are welcome in the newly forked thread headed by JoeKayza, Colonel Kernel and the others ...

I do not have any ambitious os design that is already planned in my head.
The only things that i definitely want are
-protected mode and paging
-software multitasking
-a mikrokernel approach. I dislike monolithic kernels.

There is an idea about device drivers that i have, but i neither know yet whether is possible nor have i researched on it, since i do not even know if my os will ever reach a state where i can actually think about it...

I suppose normally microkernel device drivers are using system calls for portio or for installing their interrupt vectors, and during these system calls there is a user/kernel transition which wastes some time.
I would like to reduce those system calls to two calls to associate/reserve/connect and to free/unreserve/disconnect a driver process to a single port or interrupt verktor ressource.
Then the userland driver processes could do the portio by themselves and without system calls but the concurrent access of two drivers to the same device or ports would be prohibited, because the os allows only one driver to connect to/reserve one device.
I think this might be possible using the io permission bitmap in the tss.

But we'll see... At the moment everything beyond memory management seems very far away anyway