OSDev.org

The Place to Start for Operating System Developers
It is currently Wed Aug 12, 2020 5:16 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 82 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next
Author Message
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Fri Jan 17, 2020 12:40 pm 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 743
IMO the really great invention of UNIX is not "everything is accessible via the file system" but rather "everything is a file descriptor". On the other hand, I think that Plan9's "everything is a text stream" is heavily misguided and a very bad idea. It's an oversimplification that prevents many operations from being implemented efficiently. Yes, Linux and BSDs are ugly and they need hacks like ioctl(). But that's just because the real world is ugly and complicated and not ideal and nice :).

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Fri Jan 17, 2020 3:23 pm 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 315
Apparently, Thompson & Ritchie didn't think too badly of ioctl. I was surprised when I read that.

Plan 9 performance I don't want to argue about, but I won't be really convinced until I make some tests. ;) OTOH I'd rather have proper byte-swapping than text conversion for communication on a heterogenous network. Plan 9 does some things in binary, (bitmaps, audio data, directory data, ...) mandating little-endian. *shrug*


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Sat Jan 18, 2020 12:03 am 
Offline
User avatar

Joined: Sun Dec 29, 2019 10:59 pm
Posts: 17
nyc wrote:
There are also features to consider removing outright, such as fork() as per https://www.microsoft.com/en-us/researc ... otos19.pdf or, perhaps, much of the tty/pty infrastructure in UNIX and POSIX beyond just devising better-working features instead of POSIX threading and asynchronous IO.


rdos wrote:
I don't find fork() that horrible, but it should be implemented late in the OS and after there is a CreateProcess-like interface. It's a fun challenge to get fork() right, and I don't think every odd feature of it must be supported.

I find the use of file handles for non-file objects a lot more horrible. I think this too go back to horrible UNIX hacks that are 50 years old and that should not be supported in modern OSes. I started out by refusing to support this hack, and instead users had to create a handle for the resource in question, and then need to use the functions defined for that resource-type only. So, when a a socket is created, a socket handle is returned, and it cannot be used it with write() or WriteFile(). I handle the C handle stuff simply by providing read/write (and a few other) methods for the resource-types that could be associated with C handles. This way I keep the object separation intact, but still support these horrible hacks. In fact, I think this way of handling C handles make fork() a lot more simple. Fork just duplicated the C handle translation table for the process which is kept in kernel. Resources opened with the native API rather than as C handles could just be ignored when forking as those would not be used in POSIX code anyway.


The paper gives a stronger case for removing fork() than I can reproduce from memory. The argument involves a number of relatively unique resource-sharing demands that fork() imposes among others. Even if you bolt it on afterward, the demands have impacts they decry, though maybe you wouldn't regard as significant. The paper even discusses a case, k42, where it was bolted on afterward and the negative impacts it had on it.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Sat Jan 18, 2020 11:32 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7464
Location: Germany
FWIW, AmigaOS had a rather complete POSIX layer, ixemul library. The one thing that wasn't possible to properly emulate was fork(). Most "Amiga ports" of POSIX software consisted in replacing fork() calls with something else (and convincing upstream to accept the patch)...

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Sat Jan 18, 2020 4:44 pm 
Offline
User avatar

Joined: Sun Dec 29, 2019 10:59 pm
Posts: 17
Solar wrote:
FWIW, AmigaOS had a rather complete POSIX layer, ixemul library. The one thing that wasn't possible to properly emulate was fork(). Most "Amiga ports" of POSIX software consisted in replacing fork() calls with something else (and convincing upstream to accept the patch)...


That was in some respects a limitation in only having a single address space. Not having a potential COW sharing group of fork()'s between every pair of new address spaces created by exec()'s is a very different sort of invariant from such a serious limitation in process isolation as lacking multiple address spaces. I'm buried enough in Unix spaces that I can't really give a good example of a kernel that has refused to support fork(); even Windows NT has adopted something akin to fork() for POSIX compatibility. I'm not sure about research kernels because they, too, often compromise to POSIX for the sake of getting a user application base.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Mon Jan 20, 2020 5:46 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7464
Location: Germany
I'd rather put it on it's head, for the sake of this argument: fork() makes very specific demands on how the OS manages its memory / memory spaces. And when a huge percentage of fork() uses is just to immediately follow up with exec(), or because the programmer hasn't gotten the hang of threads yet, that makes fork() the crutch in my book, not the OS not supporting it.

A lot of POSIX is quite fine as a basic foundation. When it doesn't matter either way, go with POSIX instead of rolling your own. But not every single thing in POSIX is necessarily a good idea. So if you know what the problem with a particular corner of POSIX is, you are absolutely positive you've got a better approach to it, and you can live with the limited portability of the plethora of OSS out there resulting from that deviation, by all means implement it that way.

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Mon Jan 20, 2020 10:32 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 743
I don't think implementing fork by itself is so terrible, it's CoW that is annoying (or rather, it's interaction with eviction, locking and TLB shootdown. And you probably want CoW even without fork, e.g. to implement texture buffers that transparently reside on the GPU or CPU (this is what mainstream OSes do to share the GPU among processes).

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Mon Jan 20, 2020 10:38 am 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7464
Location: Germany
...and what if you cannot, or do not want to, include address virtualization? Not all the systems of the world are desktops or servers. And fork() / COW becomes damn hard to do if you're working with physical addresses. ;-)

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Mon Jan 20, 2020 10:18 pm 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 315
Indeed. ucLinux -- Linux for systems without MMU -- didn't have fork() if I remember right. (I'm pretty sure I do.)

I'm surprised to find Minix has fork(). That page is for Minix 2.0.0. They didn't start using the MMU until after 3.0. I wonder if they copy the entire process memory?

I wonder what early Unix did.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Tue Jan 21, 2020 12:35 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 743
Solar wrote:
...and what if you cannot, or do not want to, include address virtualization? Not all the systems of the world are desktops or servers. And fork() / COW becomes damn hard to do if you're working with physical addresses. ;-)

... then you're a very specialized OS, probably not targeting the desktop, server or mobile market. You probably do not have any graphics at all (embedded) and/or only run a single task (game consoles, but is that still true these days?) and so the recommendation does not apply to you. ;-)

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Tue Jan 21, 2020 3:19 am 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 315
Korona wrote:
... then you're a very specialized OS, probably not targeting the desktop, server or mobile market. You probably do not have any graphics at all (embedded) and/or only run a single task (game consoles, but is that still true these days?) and so the recommendation does not apply to you. ;-)

I don't want to disagree with this when I think about it practically, considering modern hardware, but my inner rebel wants to spray-paint "Amiga" all over it. :twisted:


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Tue Jan 21, 2020 6:15 am 
Offline
Member
Member

Joined: Tue Jan 02, 2018 12:53 am
Posts: 44
Location: Australia
Korona wrote:
Solar wrote:
...and what if you cannot, or do not want to, include address virtualization? Not all the systems of the world are desktops or servers. And fork() / COW becomes damn hard to do if you're working with physical addresses. ;-)

... then you're a very specialized OS, probably not targeting the desktop, server or mobile market. You probably do not have any graphics at all (embedded) and/or only run a single task (game consoles, but is that still true these days?) and so the recommendation does not apply to you. ;-)

Not neccessarily! Single Address Space Operating System designs, for instance, deliberately reject address virtualization in favor of granting processes direct access to regions of physical memory, which now makes perfect sense in the 64-bit era and has many advantages over conventional designs.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Tue Jan 21, 2020 11:13 am 
Offline
Member
Member

Joined: Thu May 17, 2007 1:27 pm
Posts: 743
If you disregard spectre.

My opinion on virtual memory actually shifted into the other direction. Ten years ago I would have agreed with you (and indeed, I have a managed OS based in an in-kernel JIT on Google Code). Since then, I actually learned to like virtual memory; it enables so many useful things that stretch well beyond process isolation. Resource sharing on GPUs and accelerators, VM migration, I/O isolation via VT-d, really fast specialized userspace memory allocators, ...

_________________
managarm: Microkernel-based OS capable of running a Wayland desktop (Discord: https://discord.gg/7WB6Ur3). My OS-dev projects: [mlibc: Portable C library for managarm, qword, Linux, Sigma, ...] [LAI: AML interpreter] [xbstrap: Build system for OS distributions].


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Tue Jan 21, 2020 12:23 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 559
Most importantly, it decreases fragmentation losses. Some time ago at work we had a system running with OS-9 (no virtual memory). On the system there were only 5MB of RAM still free, but that wasn't contiguous. There was a 3MB chunk here and a 2MB chunk there. Then a process wanted to allocate 4MB. It was denied. With virtual memory, this wouldn't have happened.

OK, it turned out, with good software engineering this wouldn't have happened, either, since the 4MB allocation was only needed to load a file into memory en bloc, and we could have done what was needed in another way needing less memory, but sometimes algorithms legitimately need a lot of space.

_________________
Life is beautiful. Nobody said easy.


Top
 Profile  
 
 Post subject: Re: To POSIX or not to POSIX
PostPosted: Tue Jan 21, 2020 6:23 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 802
eekee wrote:
I'm surprised to find Minix has fork(). That page is for Minix 2.0.0. They didn't start using the MMU until after 3.0. I wonder if they copy the entire process memory?
Not all of it, just the data and stack segments. http://www.minix3.org/doc/AppendixB.pdf page 744, function do_fork(). Sorry it's pdf, ridiculous how difficult it is to find old Minix source code online (not tarballs). I was always redirected somehow to the Minix3 source...
eekee wrote:
I wonder what early Unix did.
Same, copied the segments (if you consider v5 early. The first edition is written in PDP Assembly, not C, so there's not much point in linking that). https://github.com/dspinellis/unix-history-repo/blob/Research-V5/usr/sys/ken/slp.c#L298. FYI, this is the same file with the famous "You are not expected to understand this." comment in unix-v6 btw :-)

Cheers,
bzt


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 82 posts ]  Go to page Previous  1, 2, 3, 4, 5, 6  Next

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group