Migrating threads as IPC

[kimplul]

Hi, I've been writing a kernel as a hobby project for some years now which uses thread migration as the primary form of inter-process communication (IPC). There doesn't seem to be a lot of information on the topic online, and I could only find one previous thread that brought up something similar to it: viewtopic.php?t=52225 I'd like to try stir up some discussion about the topic in general and about my approach in particular.

I'll start with a short background on the topic for context. I'll assume some familiarity with message passing, as that's generally what thread migration is compared against.

Background

Thread migration in the context of IPC means allowing threads to move between different processes, taking along some amount of data. For example, you could have a filesystem running as a userspace process, and when a thread wants to write to a file, it migrates into the filesystem, performs the write on its own (respecting other potentially concurrent threads) and migrates back to its original process. The filesystem process doesn't necessarily have to contain any threads at any given time.

Compared to a typical synchronous message passing implementation, thread migration can be more easily parallel on the service side, as multiple threads can enter a service process (filesystem, etc.) in parallel and start providing services to 'themselves'. A message passing implementation would likely have to queue all the incoming requests into some kind of mailbox, which the service process would have to fetch from and potentially distribute the messages across threads to achieve similar parallelism.

Thread migration can be seen as a procedure call into another address space, and processors are pretty good at doing procedure calls, thus thread migration can potentially benefit significantly from hardware acceleration, both on research hardware (https://dl.acm.org/doi/10.1145/3064176.3064197) and even conventional (server-grade) hardware (https://faculty.cs.gwu.edu/gparmer/publ ... 5janus.pdf). However, even with bog-standard modern hardware, thread migration is roughly as performant as message passing. The second link has a benchmark that shows unaccelerated thread migration to be roughly on par with seL4 message passing (Table 1, Composite).

I also think some soft benefits exist, such as easy userspace thread scheduling and a more 1:1 relationship between a thread and the resources it uses, which can make scheduling threads a bit easier, but these are a bit more difficult to demonstrate, so take them as opinion and with a grain of salt.

The main downside, as I see it, is that some of the flexibility of message passing is lost. For instance, message passing can potentially be used across a cluster of machines, and can allow both efficient synchronous and asynchronous communication on the same system. Thread migration is purely synchronous and realistically only applicable for communication within a single machine.

History/current state

Message passing in itself is not new, the first implementation I could find was from 1994, an experimental PA-RISC port of the Mach kernel retrofitted thread migration into an existing remote procedure call (RPC) mechanism that was previously implemented on top of message passing: https://www.usenix.org/legacy/publicati ... s/ford.pdf According to the paper, the thread migrating RPC was ~5x faster than the message passing implementation while only requiring a bit under half the source lines of code. Mach itself never really went anywhere, and the PA-RISC port in particular more or less completely disappeared, but you can still get the sources from https://www.emulab.net/downloads/OLD/pa ... ot3.tar.gz. GNU Hurd is famously based on Mach, but doesn't include the thread migration parts.

There seems to only be one active academic research project related to thread migration, which is the COMPOSITE kernel: https://faculty.cs.gwu.edu/gparmer/publ ... pert10.pdf COMPOSITE targets real-time systems, and as such is not quite usable as a general-purpose OS. In particular, it handles thread stacks in a kind of interesting way: Each process that wants to host threads migrating into it must provide them with stacks. When a thread migrates into a process, the first thing it does is check if there are any stacks available, and if there aren't it allocates one. This works pretty well when you have control over more-or-less the whole system, but means that worst-case, M*N stacks need to be allocated, where M is the number of processes and N is the number of threads.

Sun's Doors or the "process hopping" presented in viewtopic.php?t=52225 are not quite what I would understand to be thread migration, as (IIUC) they create and destroy threads on the fly. I want to stress that with thread migration, the same thread is always active and snakes its way through different processes, possibly switching or allocating stacks on the fly. Regardless, they're close enough to deserve a mention.

In short, as far as I can tell, there's currently no operating system/kernel that targets 'general usage' with thread migration.

Targeting 'general usage'/my kernel

My kernel can be found over at https://github.com/Kimplul/kmi. It currently only supports 64bit RISC-V, and it's probably closest to a hybrid kernel. I implemented memory handling within the kernel instead of outside it, mainly because I'm more interested in playing around with this cool IPC mechanism than I am about implementing a 'proper' microkernel, and having memory management in-kernel seemed a bit easier. I am semi-actively working on a unix-like userspace for the kernel (not very creative, I know), but it's not really far enough along that I'd be comfortable publishing it yet, sorry. I'm targeting 'general usage', by which I mean that the workload is unknown ahead of time and processes generally don't have full knowledge of what other processes might be running on the system and have to dynamically figure things out.

Right off the bat, each thread is allocated its own virtual address space, that is split into three regions: process, stack and kernel. The kernel region is reserved for supervisor/kernel use, fairly standard. The process region is closest to a typical userspace, all threads in a process share the same process region. The stack is region is accessible from userspace but stays static for each thread across different processes. So, when a thread migrates, it performs a system call into the kernel, where the kernel copies the target process' region mapping into the thread's process mapping and pushes an activation record onto the stack region. The stack pages accessed by the previous process are marked inaccessible for security reasons, but when the thread enters the new process, it can directly continue using the free space in the stack. This avoids the worst-case of N*M stacks, but of course means that threads can't share data on their respective stacks between each other by default. I anticipate that processes that need this behaviour (client processes like cat/grep/etc.) can be allocated stacks from the process region by libc, and service processes (filesystem server, drivers, etc.) can be written with this limitation in mind. See attachment one, where boxes are memory pages of the thread stack. Colored boxes are marked inaccessible from the current process after a thread has migrated into it. `k` is kernel data is pushed to the stack, `a`, `b` and `c` are process stack data.

Having each thread migration event be an activation record on a stack lends itself naturally to a request-response mechanism, and I've implemented a couple different 'request' kinds for better control over the control flow. Probably the simplest request is just called `req()`, and does what you'd expect: It migrates the thread to the target process, taking four registers worth of data along with it. The kernel populates the receiver's argument registers with the four data registers as well as the process ID (`eid`) and thread ID (`tid`) of where the migration came from and who did it. This way, processes or threads can't masquerade as other threads/processes. Requests are of course recursive, so if the target process find itself needing some kind of service (logging text to a file, whatever), it can perform a `req()` itself. A request is responded to with a `resp()`, which migrates the thread back to the previous process. Four registers worth of data can be returned via `resp()`, and the kernel populates a `status` register as well as the process ID of the process that responded. See attachment 0.

Let's consider a 'router' of sorts, which is a service process that just knows all other service processes currently running. A client process might not know which process ID maps to which service, and can perform a migration to the router to ask for the information (I'm assuming that the 'router' ID is always known, process 1 or whatever). This can be done by the above `req()` call, or alternatively, the client just performs the request it would've done to the service process to the router, and the router then 'forwards' the request to the appropriate process. This can be done with `fwd()`, which only differs from `req()` in that the kernel sets the process ID to the same value as when migrating to the router, so to the service process it appears as if the client performed a migration directly to it. I use the term 'effective ID', `eid`, to represent this, sort of "who this request should serve", which may or may not be the previous process. If the router process doesn't have any reason to process the service response further, it can also perform a `kick()` ('kick the can down the road'), which both performs a forwarding of the effective ID and re-uses the current activation record on the stack (kind of like a tail call in a programming language), meaning that when the service responds, the thread is directly migrated back to the client, saving the need to perform an extra migration. See attachment 2

(Hopefully the above is not too messy, I would've liked to use a separate graph for `kick()` and `fwd()` but there's an attachment limitation of 3)

I also implement signaling via thread migration. When a thread receives a signal (and it's currently in its 'root' process, i.e. where it was spawned) it is interrupted and performs a migration into the same process it's currently in. When the signal handler is finished, it 'responds' to the 'request' and the thread state is restored from the stack. Note that unlike other thread migration implementations, I perform all migrations to `_start`, so all processes are kind of 'service processes' and we can do fun stuff like the above.

I've benchmarked my kernel against seL4 and found it to be (eerily) similar in performance on real RISC-V hardware. Unfortunately not quite the same hardware, see details in my blog post below if you're curious. I have some performance optimizations and tricks that I use to avoid doing as much work as possible, and for example copying the process data region during a migration is generally a single load+store pair of overhead. I haven't gone into them in detail this post to at least attempt brevity.

There are of course a myriad of other small things that together build up to this whole thing, but I hope the above paints a good enough picture. I have a blog post about the kernel that more or less goes through the above, but maybe in slightly more detail (and with more pictures) if anyone is curious: https://metanimi.dy.fi/blog/kmi/

Discussion

I don't really have any pre-written discussion questions, but any and all feedback/questions/comments regarding the design I came up with and thread migration in general with welcome. I hope there was at least something interesting in this post and that it wasn't too difficult to follow, thanks for reading

[Octocontrabass]

The main downside, as I see it, is that some of the flexibility of message passing is lost. For instance, message passing can potentially be used across a cluster of machines, and can allow both efficient synchronous and asynchronous communication on the same system. Thread migration is purely synchronous and realistically only applicable for communication within a single machine.

You can still do asynchronous operations by starting a new thread for each asynchronous operation. I'm not sure if that's better or worse than "regular" asynchronous message passing.

How does this work for something like a disk cache, which might want to asynchronously read ahead from slow storage media or asynchronously write modified files back to disk?

How does this work in drivers for modern high-performance hardware like NVMe, where the hardware itself uses asynchronous message passing?

The stack pages accessed by the previous process are marked inaccessible for security reasons,

Why migrate the caller's stack if it won't be accessible? Or is the idea that some processes will be trusted enough to access the caller's stack?

This brings to mind a hybrid approach where instead of migrating the entire thread, you spawn a new thread in the target process that gets scheduled in place of the calling thread. Since you'd be starting a new thread anyway, the only difference between synchronous and asynchronous operations would be whether or not the calling thread is blocked until the new thread exits.

[kimplul]

You can still do asynchronous operations by starting a new thread for each asynchronous operation. I'm not sure if that's better or worse than "regular" asynchronous message passing.

Yep, given powerful enough primites one can build other kinds of communication on top. Message passing can be implemented on top of thread migration and vice-versa, it just boils down to how efficient the primitives are at performing certain kinds of operations. I just meant that to get async communication, you have to implement it on top of synchronous operations, as that's all you get with thread migration.

Although, I would argue there are likely more efficient ways to get async communication than spawning threads, at least with my kernel. Since each thread lives in its own virtual address space mapping, creating a thread is a fairly expensive operation, and you'd likely be better off by implementing some kind of message passing within each process that wants to use async communication. So this would in practice maybe be something like a thread performs a migration into the service process, the service process places request into a queue and the thread returns back to the original process. A separate worker thread owned by the service process then handles the requests in the queue and signals the thread whose request is finished to come pick it up.

Alternatively, some kind of io_uring-like approach where requests are just placed in a shared memory buffer, again with at least one worker thread.

How does this work for something like a disk cache, which might want to asynchronously read ahead from slow storage media or asynchronously write modified files back to disk?

How does this work in drivers for modern high-performance hardware like NVMe, where the hardware itself uses asynchronous message passing?

I guess that would depend on how the disk cache/NVMe driver is structured moreso than the IPC method by which the driver receives requests, unless I'm misunderstanding the question? Some kind of queue of file operations is perfectly fine to use within the driver itself, the parallel nature of thread migration is not particularly useful when interacting with individual devices that require some kind of sequencing anyway.

However, the neat thing with thread migration is that serializing requests can be done on a per-need basis, instead of being required by the kernel. So in the case of the disk cache, several threads could potentially access the cache in memory in parallel and that way get more performance (assuming a suitably parallel data structure etc), whereas with a 'typical' (for some definition of typical) message passing implementation, the cache would either serve one request at a time or have to unserialize requests, perform the accesses in parallel and then reserialize the responses.

Why migrate the caller's stack if it won't be accessible? Or is the idea that some processes will be trusted enough to access the caller's stack?

Poorly explained on my part, sorry. Only the pages touched by the previous process are marked inaccessible, the rest of the stack is available for use in the next process. Not sure I would even say that the stack is 'migrated', since it's statically mapped into the thread's address space.

To be more specific, I currently reserve one top-level page table entry for the stack region, and then map a 2MiB stack in 4KiB pages into that region such that they trap on first access. That way, when a migration requested, I can quickly check what the highest stack address accessed was and calculate how many pages to mark invalid, and iterate over them in a tight loop. Nothing else is done to the stack. See attachment 0 for how this looks with Sv39:

When returning back to the process where the thread left from, the kernel makes the pages accessible again and sets them to trap on first access so that the 'current' stack usage of the process is used when the next migration occurs, instead of marking pages based on the maximum stack usage.

This brings to mind a hybrid approach where instead of migrating the entire thread, you spawn a new thread in the target process that gets scheduled in place of the calling thread. Since you'd be starting a new thread anyway, the only difference between synchronous and asynchronous operations would be whether or not the calling thread is blocked until the new thread exits.

Sure, at least assuming spawning a thread is a reasonably quick operation. I imagine you'd want to keep around a cache of stacks and so on to speed it up, which is probably not a huge issue but is something extra to worry about. With my scheme, you allocate one stack for the thread which can then be used across processes, which at least to me seems like the easier approach.

Although I suspect that having a single stack per thread might have some performance benefits compared to creating a new thread. It at least helps with cache locality, since a 'static' thread stack is more likely to have been recently accessed than a stack that was just pulled in from somewhere. A thread-specific stack is also more likely to not have its cachelines be shared, which decreases cache coherence traffic. Also, not having to check a cache on each IPC operation is at least one less branch to mispredict. However, I don't have any hard numbers to back this up, so take this with a grain of salt.

[Octocontrabass]

Only the pages touched by the previous process are marked inaccessible, the rest of the stack is available for use in the next process.

So you aren't migrating the stack, you're creating a new stack but reserving the address space where the caller's stack was located to speed up thread migration.

When returning back to the process where the thread left from, the kernel makes the pages accessible again and sets them to trap on first access so that the 'current' stack usage of the process is used when the next migration occurs, instead of marking pages based on the maximum stack usage.

Why bother with traps at all? The stack pointer tells you the current stack usage.

[VincentVL85]

I feel like I'm having trouble understanding the basic operational theory at the level of abstraction you're using (no insult intended, I call my own project MAD, I know the feeling of trying to explain a jungle in a few paragraphs). On your blog you talk about starting the process at _start(...); does that imply that when you join a process, you are effectively calling an entrypoint function in the target process space, and the entrypoint function looks at the parameters and stack that got passed in to decide what to do next? Is there one entrypoint per process, or is one of the parameters being passed in a target function to be called? Or does the calling function get control back so it can execute arbitrary code in some new process space, which seems unlikely?

I like to think about IPC/RPC as a high-level construct, but I'm admittedly not nearly familiar enough with how it works on the lowest levels, so I'm having trouble pinning exactly where this belongs in the picture of, eg, "library X exposes API function Y() which calls native function Z() inside the process space, the code for Y is stored in a shared object file and dynamically linked at runtime".

[kimplul]

So you aren't migrating the stack, you're creating a new stack but reserving the address space where the caller's stack was located to speed up thread migration.

I suppose? I think I'm just getting hung up on the wording used, but just to hammer it home for future readers: The whole stack is created and mapped in 4KiB chunks when the thread is created, and only then. The only thing that happens during a thread migration is that some of the pages in this stack are marked unavailable (and then correspondingly marked available again when returning).

This stack lives in its own special region that is available only to the thread to which the stack belongs, so I don't think it really makes sense to say "where the caller's stack was located", as the stack itself is always at the same virtual address for all threads. The process that is currently using the stack is just not allowed to access certain parts of it, depending on how much of the stack has been used by previous processes in the migration chain.

When returning back to the process where the thread
Why bother with traps at all? The stack pointer tells you the current stack usage.

In case the process decides to use another stack, for example because of stateful coroutines or because it tries to provide a POSIX-like environment where threads can access each other's stacks. If I just looked at the stack pointer, it could point outside the thread-specific stack in the above cases and I wouldn't know which parts would need invalidation.

[kimplul]

I feel like I'm having trouble understanding the basic operational theory at the level of abstraction you're using (no insult intended, I call my own project MAD, I know the feeling of trying to explain a jungle in a few paragraphs).

None taken I've been in the headspace of thread migration for a good while now, I think I've just forgotten what parts are obvious and what parts aren't.

On your blog you talk about starting the process at _start(...); does that imply that when you join a process, you are effectively calling an entrypoint function in the target process space, and the entrypoint function looks at the parameters and stack that got passed in to decide what to do next? Is there one entrypoint per process, or is one of the parameters being passed in a target function to be called? Or does the calling function get control back so it can execute arbitrary code in some new process space, which seems unlikely?

It's essentially the same interface as when performing a regular system call in a monolithic kernel, except that the request is directed to another process. In a monolithic kernel, the kernel has one entry point (for syscalls, at least), and the kernel then looks at the given register arguments to decide what to do. In practice you'd probably want to reserve one of the registers for passing in an enum that decides the operation, I guess you could also just directly use a pointer but that'd be a bit unsafe The calling process (it's the same thread in both the caller and callee, to be pedantic) does not execute arbitrary code, just code that the callee's developer has specified.

Something like the following example:

Caller:

Code: Select all

sys_call(IPC_REQ, SOME_PROCESS_ID, SOME_TARGET_OP, ARG0, ARG1, ARG2);

Callee:

Code: Select all

void _start(reg_t eid, reg_t tid, reg_t op, reg_t a0, reg_t a1, reg_t a2)
{
	/* permission checks or similar, migrates back with error code if an issue is found */
	check_requester(eid, tid);
	
	/* assuming each branch responds to the request on their own */
	switch (op) {
	case SOME_OP: do_some_op(eid, tid, a0, a1, a2);
	case SOME_OTHER_OP: do_some_other_op(eid, tid, a0, a1, a2);
	default:
	resp_error(ERR_UNKNOWN_OP);
	}
}

At least one branch should be reserved for handling kernel-triggered operations, like creating a new thread in the address space (potentially other stuff in the future, like a request to dump caches when memory is low etc.)

I like to think about IPC/RPC as a high-level construct, but I'm admittedly not nearly familiar enough with how it works on the lowest levels, so I'm having trouble pinning exactly where this belongs in the picture of, eg, "library X exposes API function Y() which calls native function Z() inside the process space, the code for Y is stored in a shared object file and dynamically linked at runtime".

I'm sorry, I'm not sure I fully understand your question, but client processes would presumably use libraries that wrap around this IPC mechanism. For example in libc you'd have `fopen()` which would in this case be implemented as something like

Code: Select all

int fopen(const char *path)
{
	/* skipping safety checks/mutexes/etc, assuming a shared memory region
	 * has been previously opened between this process and the process
	 * responsible for the filesystem */
	memcpy(shared_memory, path, strlen(path) + 1);
	
	/* assume the filesystem process knows to look for the path in the shared memory */ 
	struct sys_resp resp = sys_call(IPC_REQ, FS_PID, FS_OPEN);
	
	/* IPC request itself failed (not enough stack memory, process doesn't exists, etc) */
	if (resp.status != 0)
		return -1;
		
	/* filesystem process itself responded with an error
	 * (assume some convention of the a0 register being a status reg) */
	if (ret.a0 != 0)
		return -1;
	
	/* actual file descriptor is in a1 */
	return ret.a1;
}

Does that clear anything up or am I barking up the wrong tree here?

[VincentVL85]

Ok, that all makes sense. Or rather, it makes as much sense as system calls, and frankly, part of my confusion is I don't really understand or like system calls, to the point where I wish device drivers weren't. That's... kind of my own hangup, one of many tied in with my whole project.

The whole thing about libraries is to say, that generally a service would want to advertise an API for interacting with it, complete with all the nuances of types, error conditions, and so on, and ideally, the client (library) and service would be subject to the same static analysis (eg, header files) so that you know they always perfectly match. If a high level language construct gets wrapped around this thread migration IPC, then you can certainly pass arbitrary arguments on the stack - but for the static analysis to work well, you would want to be able to guarantee, within the higher level language, that arguments passed to library function Y are passed to service function Z unchanged, because if there were any side effects in the process, sooner or later someone would fail to account for them and it'd be a mess.

Anyway, thanks for explaining. I do feel like it has its uses, and I had not heard of it before.

[Octocontrabass]

I suppose? I think I'm just getting hung up on the wording used,

Possibly. I'd say it's a single memory region containing multiple stacks, since they're completely isolated from each other.

In case the process decides to use another stack, for example because of stateful coroutines or because it tries to provide a POSIX-like environment where threads can access each other's stacks. If I just looked at the stack pointer, it could point outside the thread-specific stack in the above cases and I wouldn't know which parts would need invalidation.

You could define the ABI so that threads are required to set the stack pointer to the top of the thread-specific stack before attempting to perform IPC. This adds a small amount of complexity to some userspace programs in exchange for making thread migration much simpler for the kernel (and possibly faster, since there are no traps, and possibly more reliable, since there's no chance that the kernel accidentally guesses the stack is empty when a thread performs two IPC calls back-to-back without touching its stack between them).

[kimplul]

You could define the ABI so that threads are required to set the stack pointer to the top of the thread-specific stack before attempting to perform IPC. This adds a small amount of complexity to some userspace programs in exchange for making thread migration much simpler for the kernel (and possibly faster, since there are no traps, and possibly more reliable, since there's no chance that the kernel accidentally guesses the stack is empty when a thread performs two IPC calls back-to-back without touching its stack between them).

That could be a good approach, thanks for the suggestion! Handling stack traps was fairly easy to set up since the kernel has to handle segfaults etc. anyway and the trap overhead hasn't been a significant concern at least yet, since the stuff I've been playing around with has had fairly low stack usage, rarely exceeding a page.

For reliability, the edge case of not touching the stack is currently handled by restoring the previous max stack usage. If no pages are touched, the same area is marked again on the next migration. Your suggestion would make that particular variable redundant, saving a load-store pair Although I imagine the userspace ABI handling would be a bit more expensive, but probably not by a lot.

[VincentVL85]

So, I'd like to take you up on the offer to discuss this. Feel free to take everything I say with however many grains of salt you think my reputation deserves, and sorry if this ends up being any kind of headache.

Background: MAD is a hypothetical operating system and programming methodology combo that is designed to make it easier to combine multiple pieces of hardware into a gestalt system. This requires, among other things, a lot of IPC/RPC and userland device drivers (at least the high level drivers).

For our purposes here I'm going to define the word “application” to be, basically, any program that is expected to hit a generic event-wait loop, whether that's waiting on user input, or waiting for signals from another program, etc. All services are apps, and all (high-level) drivers are services. Programs that never hit that generic event loop are not apps no matter how long they run (because they can't respond to general events). Pretty much all programs with a GUI are apps because they have to check for user input, GUI events, etc.

Part of the complication of understanding thread migration (and similar things) is how the words “process” and “thread” sound like they mean the same thing, especially in a context where you have to distinguish between them. It makes intuitive sense to me that, for example, that a running application (ongoing program) can be treated like a dynamic library if it advertises itself as such, and when another source application links to it, the running thread ducks into the target application's code space temporarily. It's intuitive enough that I just kind of assumed it without really thinking about it. (If it sounds like I'm describing something different, by all means correct me)

The primary advantage for me specifically, to linking directly to a running app, is not needing to use a shared library object file in order to interface with a service (or driver). I want to standardize interfaces (in the sense of APIs), which means using one identifier to stand in for a specific collection of exposed functions, which another app could link to. If a running service claims to implement an interface, you need neither a library nor a vendor-specific header file (during programming) to make the connection. The generic interface header claims that certain functions will be available, and then the service provides metadata on where to find those functions at link/call time.

For instance, I want to abuse the filesystem metaphor to make app-internal data available as file nodes, and that makes sense as long as the running application presents itself as a file server. In my mind, file servers should act like (look like/claim to be) filesystem drivers, and drivers are services are apps; it's a tautology that, if drivers are userland apps, all apps can act like drivers as well. They just expose functions and advertise themselves as linkable, and the OS takes them at their word. For this example, the OS would collect all running filesystem drivers and organize them, and any program that does this makes its internal data available as files.

It's really more complicated and more difficult, yes, but architecturally? Architecturally, if you can treat running apps like shared objects and link to them, then you open up a whole world.

Now, you said that thread migration is less appropriate for remote communications, and I get what you're saying, but does my logic follow?

• Thread migration, does/can mean linking to services like object files
• You have to write all apps that can be linked to, in such a way that they have thread-safe memory access (in other words, this should be standard)
• The actual migration should be done by a kernel function that can reliably handle errors, crashes, and other abnormalities; once centralized, we can expand its role
• If you want to “migrate a thread” to another machine, you really want to migrate to a network handler that sends out a message and waits for a reply. This handler should also be standard and safe, and handle timeouts and network events, but once it is done, we can expand its role too.
• An application that is waiting on network events is no different than an application waiting on any other user input; it simply enters that generic event loop, with a bookmark saying it expects to hear back from so-and-so. (This is important architecturally, moreso than specifically here)
• A service doesn't care whether the calling application is on the same machine or not (unless it has specific reason to). Having a thread migrate “in” from a network handler is no different from having it migrate in from an app.

Thus if you write programs with thread migration in mind, it's entirely reasonable to chop off half of an application and put it on another computer, as long as the program logic is made contiguous (including memory management). Intercept a function call, turn it into a network request and wait. On the far side, open a new thread, migrate that thread, wait on the result, send the reply and close out.

This is admittedly not optimization-oriented. But what I'm looking for are models that do the best job of being a generic model. A system that only assumes message passing and never allows anything like thread migration, needs shared libraries to fulfil the same purpose.

Am I making sense? Is there anywhere I'm obviously wrong?

[kimplul]

Part of the complication of understanding thread migration (and similar things) is how the words “process” and “thread” sound like they mean the same thing, especially in a context where you have to distinguish between them. It makes intuitive sense to me that, for example, that a running application (ongoing program) can be treated like a dynamic library if it advertises itself as such, and when another source application links to it, the running thread ducks into the target application's code space temporarily. It's intuitive enough that I just kind of assumed it without really thinking about it. (If it sounds like I'm describing something different, by all means correct me)

Apologies if I'm being patronizing, but it sounds like drawing a clear separation between processes and threads is in order.

Both 'processes' and 'threads' evolved from simpler hardware-specific features. Initially, computers only had a single processor and could only run a single program. At some point, people thought "hey, wouldn't it be cool if we could run multiple programs", and the winning solution was to essentially create small isolated 'virtual computers' that each program could run in. This is what a process is, just a context within which to run some code, separated from other bits of code so that a bug in one process can't mess up the execution context of another bit of code. When computers started getting multiple cores, it was kind of natural to say "well, since a process is a kind of virtual computer, why not give them multiple 'cores' as well", which is what threads are.

'Process' = 'Virtual computer' is maybe not a perfect analogy, since processes can generally rely on an operating system to provide high-level services, which a program running on bare hardware can't do, but hopefully it draws a clear(-er) distinction between a thread and a process.

Thinking of processes as libraries is not unheard of by any means, but has the complication that since each process is it's own little 'computer', you can't just willy-nilly pass pointers to data around like you can within a process. That's where Inter-Process Communication (IPC) comes along and provides some structured way to pass chunks of data across these 'computers', kind of like how real computers have to send TCP/IP packets around via real cables. Switching between processes is also (with conventional hardware, anyway) always a lot slower than just doing a library call, since the hardware has to switch execution contexts, which can take thousands of cycles (compared to tens for a procedure call). Even worse if you're trying to communicate between two physical computers, where latencies can rise to millions of cycles.

Personally, I find it kind of unfortunate that we're still essentially running really beefed up PDP-11's as our computers. I think we need more unconventional hardware, which could also spice up operating system development quite a bit, potentially more in the distributed direction. There's a good keynote on it by Timothy Roscoe, one of the developers of the (discontinued) Barrelfish operating system: https://www.youtube.com/watch?v=36myc8wQhLo

I've been toying around with some ideas around accelerating fine-grained parallelism and I have some amateurish RTL stuff, but that's not strictly relevant and probably a topic for another time

The primary advantage for me specifically, to linking directly to a running app, is not needing to use a shared library object file in order to interface with a service (or driver). I want to standardize interfaces (in the sense of APIs), which means using one identifier to stand in for a specific collection of exposed functions, which another app could link to. If a running service claims to implement an interface, you need neither a library nor a vendor-specific header file (during programming) to make the connection. The generic interface header claims that certain functions will be available, and then the service provides metadata on where to find those functions at link/call time.

For instance, I want to abuse the filesystem metaphor to make app-internal data available as file nodes, and that makes sense as long as the running application presents itself as a file server. In my mind, file servers should act like (look like/claim to be) filesystem drivers, and drivers are services are apps; it's a tautology that, if drivers are userland apps, all apps can act like drivers as well. They just expose functions and advertise themselves as linkable, and the OS takes them at their word. For this example, the OS would collect all running filesystem drivers and organize them, and any program that does this makes its internal data available as files.

I believe that's more or less what microkernels do. They have some well-defined APIs, and the services that implement a specific API report that to potential clients in some way. Some even do this across a network of interconnected computers: https://doc.redox-os.org/book/schemes.html

Redox strictly follows the 'everything is a file' paradigm, which is well-defined but I kind of got the impression that you were maybe talking about some higher-level services, like 'render an image' or something along those lines. I guess it could be implemented on top of files, but then you'd have to specify the exact structure of the 'file system' provided by each image renderer.

Have you heard of the M3 operating system? I think it sound kind of similar: https://github.com/Barkhausen-Institut/M3

[*]You have to write all apps that can be linked to, in such a way that they
have thread-safe memory access (in other words, this should be standard)

I'm not entirely sure what this means. If all applications live in their own processes, the code within the process can do whatever memory accesses it wants, if a segfault occurs the process is just killed and doesn't (directly, at least) affect other processes. If applications all share the same memory space, you'd have to prove that the code is memory-safe, which is an open problem generally (although you can potentially restrict programs to some provably-safe constructs, like with BPF in Linux).

Regarding the other points in the list, sure, you can absolutely implement some form of thread migration on top of message passing between computers, you just have to either create a new thread for each message (slow) or keep around a cache of idle threads (wastes some amount of memory, but maybe not too much?).

I don't fully understand what you mean by 'never allows anything like thread migration', since as you say, just have a thread react to an incoming message and now you have something that acts like thread migration. I don't necessarily think having it be part of a shared library is a huge problem, you could have in a static library or in the kernel proper, but maybe I'm misunderstanding something.

Sidenote, it's unclear to me what you mean by memory management across computers, if they don't share any memory they don't need 'coherent' memory management, and if they do share memory, you have to deal with the latency issues I mentioned above, and at that point thread migration is likely the least of your concerns. I guess a proper description of what kind of hardware you're envisioning would be useful, but that's maybe a bit off topic for this thread

[VincentVL85]

Part of me wants to be a little defensive, but I asked because I wanted to have thing clarified by people who clearly understand better than me. I am holding a bunch of complicated pieces in my head, many of which I only half-understand, and it'd be miraculous if I was right about all of it! I will look into Redox and M3. Redox, when it crossed my feeds, only sounded like “…but we did it in rust!” so I didn't really look at what else they were doing. I do like to understand what similar projects are thinking, but it's not clear how many are really relevant. I have structural disagreements with Plan 9, for instance, which may be irreconcilable. I do know that microkernels share a lot of what I'm looking for… but I'm trying to establish requirements, because testing the ideas against those requirements is how I evaluate what I'm working on. A vague idea that doesn't provide requirements or constraints isn't quite enough.

I'm not entirely sure what this means. If all applications live in their own processes, the code within the process can do whatever memory accesses it wants, if a segfault occurs the process is just killed and doesn't (directly, at least) affect other processes. If applications all share the same memory space, you'd have to prove that the code is memory-safe, which is an open problem generally (although you can potentially restrict programs to some provably-safe constructs, like with BPF in Linux).

I mean typical race condition things. The point of linking to a service is essentially that you want the service to do something and/or you want to leverage its knowledge, but it may also already be doing something that might conflict and/or its knowledge may be in flux due to nominal service threads or other IPC threads. Services don't only replace libraries; insofar as IPC threads touch anything, you have to make sure only one thread is touching it at a time. Thus, if every app can have IPC threads, every app needs to be alert for race conditions.

On the definition of processes…

I do understand virtual memory, at least in the broad strokes. Part of what I was saying is that “virtual memory” and “process” and “thread” are bad terms, and “process” and “thread” specifically are basically synonyms meaning “going through things one step at a time”. The idea that “process” as a word stands in for a specific virtual memory space and “thread” stands for an execution pointer (I am simplifying, I know) is annoying, not least because “process” sounds like an action and “thread” sounds like an object.

I'd like to think of a process space/VM/application memory as specific regions of memory that the current thread is whitelisted to access (often/always relocated, but in blocks), in which case it makes sense to me that in the case of a migrating thread, certain blocks of memory can be whitelisted if you need that data to cross process boundaries, and while the blocks may relocate during the crossover, it doesn't sound all that farfetched to say that within the same machine at least, certain memory blocks can be shared between them. You would want some rigorous methodology and well-established best practices to ensure you aren't doing something entirely reckless, but as far as I'm concerned, the central question is whether the architectural scheme makes any damn sense or whether I'm just an idiot.

Sidenote, it's unclear to me what you mean by memory management across computers, if they don't share any memory they don't need 'coherent' memory management, and if they do share memory, you have to deal with the latency issues I mentioned above, and at that point thread migration is likely the least of your concerns.

Part of my programming model is breaking programs down into context-specific (usually, device-specific) namespaces, which represent separable memory blocks (in the sense of “chop your program in half” as I said before). Because they are separable, code in one namespace has to check when accessing code or memory in another namespace, whether that namespace is local or remote. In principle, if the namespace is remote, it simply gets translated into an RPC targeting another part of the same application on another computer. (This is safer than making RPC calls against another application because the the static type analysis is consistent between local and remote versions of the call, and it's all compiled at once) “Memory management between machines” might be exactly that simple… in specific cases where programmers do everything right.

Let's say generally that when you create an IPC function you add two optional namespaces: “shared host data” and “shared caller data.” SHD happens if the service needs to set aside memory for the caller, and it should be jointly held; for example, a GPU driver that must set aside DRI memory for an application. The DRI memory is in the host's keeping, but it arguably also belongs to the caller, if only because SHD can also be the backing memory for a return value from an IPC function; it should not go out of scope while it remains in an owner's control. But also, if the calling process dies, all memory allocated under its auspices should be freed, including memory jointly held by another process. SCD on the other hand, happens whenever you want to pass in data that the calling process owns, without making a redundant copy. The host can copy SCD into SHD if they need to keep it.

Now, again, MAD is not about speed optimization; any optimization we do is to claw back losses, not get ahead. (The UG college professor that I shared my ideas with 15+ years ago was also a proponent of optimization rather than flexibility, so suffice it to say he did not encourage pursuit of these ideas.) That said, no matter how much I as a highly optimistic and naive theorist want to prevent programmers from trying to access SHD/SCD over a network, some idiot will, inefficiency be damned. It might even be necessary, in some fraction of those cases.

As you say, if we did this by sending messages back and forth for every memory read/write, the latency would be unbearable. That leaves us with the unfortunate requirement of copying and caching SCD/SHD during remote calls, and figuring out when to synchronize that data and when to free it. The alternative to caching would be… not caching. Which I guess would also work, as long as you are prepared for the consequences. I feel like it's completely inevitable, no matter what safeguards you put in, that programmers will want to maintain copies of large data blocks on multiple machines and keep them synchronized.

Even without going that far, though, what I said is that "program logic including memory management should be contiguous between machines". That means that you are always prepared to access memory that is logically part of the application but not physically present, and that memory access should be part of the application logic itself. Imagine debugging a program that is spread across multiple machines: the entire scope of the application's memory should all be available to the debugger, exactly as it would be if it were all local, but with lines drawn across it showing where the actual boundaries are.

Redox strictly follows the 'everything is a file' paradigm, which is well-defined but I kind of got the impression that you were maybe talking about some higher-level services, like 'render an image' or something along those lines. I guess it could be implemented on top of files, but then you'd have to specify the exact structure of the 'file system' provided by each image renderer.

I would not use the term well-defined to describe extant “everything is a file” models that I have looked at (this is me being a grump, and probably ignorant, rather than a scathing indictment of the industry), but let's talk about rendering an image. My thoughts echo a lot of the Redox Schema stuff you linked to, but I and they are probably not on exactly the same page.

Say the running GPU driver is represented by a process file or folder, and you link to it to access its advertised interfaces via IPC. One interface function reserves a DRI memory backplane, which is accessible as a SHD namespace. While you might think of that backplane as a raw data address, remember that you are linking directly to the driver and can use all its functions. So if the GPU has a function to render a bitmap into the DRI backplane, you only really need to pass it the handle of the backplane and send in the bitmap as SCD.

Now, what happens if the image you want to render is neither in bitmap form nor local to the machine with the GPU? Well, it's not complicated, architecturally, theoretically. Parse the image file and convert it to a bitmap (or other raw image data) either before or after you send that data to the GPU machine. The network handler caches the bitmap as SCD (or the file as SCD and the bitmap as SHD if the GPU decodes it) and then it's used as a parameter in the DRI render() function.

An “everything is a file” model would imply that the above can be described as a shell script. Something like “this/dri = gpu/dri:Create(); this/dri:Render(this/files/smile.jpg:ToBitmap(),0,0);” A well-defined "everything is a file" model should be able to make these kind of statements representing any accessible, well typed, exported function and/or data, even if the filesystem represents multiple machines networked together. In other words, shell scripts should basically be able to do anything that compiled programs can do, because everything is a file.

So yeah, I have unrealistically high standards, and I have written… a fair many words about how that is more complicated than it looks. I'm sure others who have looked at the same problem from different angles would agree. Whether we all agree on if we can/how to make that complicated problem manageable or not will take some more research on my part.

I don't fully understand what you mean by 'never allows anything like thread migration', since as you say, just have a thread react to an incoming message and now you have something that acts like thread migration. I don't necessarily think having it be part of a shared library is a huge problem, you could have in a static library or in the kernel proper, but maybe I'm misunderstanding something.

The existence of shared libraries isn't a problem. Requiring a shared library in order to do IPC with a service, is not so much a problem as it is a sidestep that I am now hoping might be unnecessary. What I'd like to do is reference some function that a service provides as an unambiguous filesystem node, which should be a child node of the service's process folder. If the function is attached to a library instead of the service itself, it may go from “service:function()” to “service/lib:function()” which is far from a touch of death. I was planning to get around this with metadata, and maybe that's still a superior solution. Or the way metadata works simply needs to shift to accommodate it. Whatever.

As far as what hardware I'm envisioning… probably better for another time. Basic processor architectures, if that's the question. Mostly, the whole design is built around the idea that you will have network nodes so minimal that they cannot do anything but host processes, without so much as disk access (aside from an immutable OS image) or USB/serial ports. The point, is that these processor-only nodes are not there only for data center scale, grid compute workloads. I want a solution that lets you borrow a processor-only node for your desktop application (probably instead of, rather than in addition to, CPU space on a more generic host computer). That requires a programming methodology that both works and makes sense to the programmer. That's what I'm chasing, and it's what I think I have, it just requires… some radical decentralization.

[kimplul]

I think this discussion is steering away from thread migration, and as such is maybe getting a bit off topic. I wouldn't mind continuing in a separate thread, though.

I mean typical race condition things. The point of linking to a service is essentially that you want the service to do something and/or you want to leverage its knowledge, but it may also already be doing something that might conflict and/or its knowledge may be in flux due to nominal service threads or other IPC threads. Services don't only replace libraries; insofar as IPC threads touch anything, you have to make sure only one thread is touching it at a time. Thus, if every app can have IPC threads, every app needs to be alert for race conditions.

Gotcha, poor reading on my part. Seems I glossed over the 'thread-safe' part and thought you were talking about memory safety specifically.

I do understand virtual memory, at least in the broad strokes. Part of what I was saying is that “virtual memory” and “process” and “thread” are bad terms, and “process” and “thread” specifically are basically synonyms meaning “going through things one step at a time”. The idea that “process” as a word stands in for a specific virtual memory space and “thread” stands for an execution pointer (I am simplifying, I know) is annoying, not least because “process” sounds like an action and “thread” sounds like an object.

Right, so the chosen words are bad, not that the concepts are unclear? I don't have any strong opinions on the matter, you get used to them I suppose

I'd like to think of a process space/VM/application memory as specific regions of memory that the current thread is whitelisted to access (often/always relocated, but in blocks), in which case it makes sense to me that in the case of a migrating thread, certain blocks of memory can be whitelisted if you need that data to cross process boundaries, and while the blocks may relocate during the crossover, it doesn't sound all that farfetched to say that within the same machine at least, certain memory blocks can be shared between them. You would want some rigorous methodology and well-established best practices to ensure you aren't doing something entirely reckless, but as far as I'm concerned, the central question is whether the architectural scheme makes any damn sense or whether I'm just an idiot.

Shared memory exists and is fairly widely used. Accessing it should be treated as accessing any memory that is shared between threads in a process, either via a mutex or some atomic data structure. Some languages provide type checking for it. I believe that more or less covers best practices, or do you mean something else?

Part of my programming model is breaking programs down into context-specific (usually, device-specific) namespaces, which represent separable memory blocks (in the sense of “chop your program in half” as I said before). Because they are separable, code in one namespace has to check when accessing code or memory in another namespace, whether that namespace is local or remote. In principle, if the namespace is remote, it simply gets translated into an RPC targeting another part of the same application on another computer. (This is safer than making RPC calls against another application because the the static type analysis is consistent between local and remote versions of the call, and it's all compiled at once) “Memory management between machines” might be exactly that simple… in specific cases where programmers do everything right.

Let's say generally that when you create an IPC function you add two optional namespaces: “shared host data” and “shared caller data.” SHD happens if the service needs to set aside memory for the caller, and it should be jointly held; for example, a GPU driver that must set aside DRI memory for an application. The DRI memory is in the host's keeping, but it arguably also belongs to the caller, if only because SHD can also be the backing memory for a return value from an IPC function; it should not go out of scope while it remains in an owner's control. But also, if the calling process dies, all memory allocated under its auspices should be freed, including memory jointly held by another process. SCD on the other hand, happens whenever you want to pass in data that the calling process owns, without making a redundant copy. The host can copy SCD into SHD if they need to keep it.

I'm not familiar with namespaces as a term in this context, so maybe I'm just missing your point but I'm not sure what redundant copy you're referring to, shared memory uses the same physical pages but maps them into two separate virtual memory regions. If you just mean that one process allocates a memory buffer and then temporarily gives it over to some other process, that's not really shared memory, that's just a form of message passing.

If the memory is shared, what happens if the calling process is killed while the called process is writing into the buffer? Is the called process also killed because it tried to use a freed buffer (since all buffers are freed when the owner dies) or is the thread that would've used the buffer killed as well? That could leave mutexes locked or other resource in indeterminate states.

Now, again, MAD is not about speed optimization; any optimization we do is to claw back losses, not get ahead. (The UG college professor that I shared my ideas with 15+ years ago was also a proponent of optimization rather than flexibility, so suffice it to say he did not encourage pursuit of these ideas.) That said, no matter how much I as a highly optimistic and naive theorist want to prevent programmers from trying to access SHD/SCD over a network, some idiot will, inefficiency be damned. It might even be necessary, in some fraction of those cases.

As you say, if we did this by sending messages back and forth for every memory read/write, the latency would be unbearable. That leaves us with the unfortunate requirement of copying and caching SCD/SHD during remote calls, and figuring out when to synchronize that data and when to free it. The alternative to caching would be… not caching. Which I guess would also work, as long as you are prepared for the consequences. I feel like it's completely inevitable, no matter what safeguards you put in, that programmers will want to maintain copies of large data blocks on multiple machines and keep them synchronized.

Even without going that far, though, what I said is that "program logic including memory management should be contiguous between machines". That means that you are always prepared to access memory that is logically part of the application but not physically present, and that memory access should be part of the application logic itself. Imagine debugging a program that is spread across multiple machines: the entire scope of the application's memory should all be available to the debugger, exactly as it would be if it were all local, but with lines drawn across it showing where the actual boundaries are.

I'm not sure you can keep them synchronized (transparently) without significant latencies. Caching is well and good for reads, but if even a single byte is written, it needs to be sent over the network, and the other nodes on the network will be reading the old value from their caches until the new value arrives. I suspect that explicit synchronization points would be required at minimum, something like barriers in OpenCL.

Still, it's not obvious to me why the application's memory should be shared across computers at all. If the point is that different nodes implement different services, why can't I just send messages to these services and let each node handle its own memory? If I need to debug an issue, why not just ask the service to send me back the data that I'm explicitly interested in? The 'idiot' can't access SHD/SCD's over a network if you don't provide that option.

I would not use the term well-defined to describe extant “everything is a file” models that I have looked at (this is me being a grump, and probably ignorant, rather than a scathing indictment of the industry), but let's talk about rendering an image. My thoughts echo a lot of the Redox Schema stuff you linked to, but I and they are probably not on exactly the same page.

I suspect we're talking of slightly different abstraction levels. From my perspective, a file provides the operations open (you get a new handle)/read (you get stuff out)/write (you put stuff in)/close (you ditch the handle), which is about as well-defined as can be I guess you're more interested in higher-level interfaces implemented via files...?

An “everything is a file” model would imply that the above can be described as a shell script. Something like “this/dri = gpu/dri:Create(); this/dri:Render(this/files/smile.jpg:ToBitmap(),0,0);” A well-defined "everything is a file" model should be able to make these kind of statements representing any accessible, well typed, exported function and/or data, even if the filesystem represents multiple machines networked together. In other words, shell scripts should basically be able to do anything that compiled programs can do, because everything is a file.

Sorry, I don't quite follow. I don't see how 'everything is a file' (which is a bit hyperbole, to be clear) would imply that operations on files should be well typed. You can build tools that add typed frameworks on top of files, something like NuShell, but files themselves are just bags of bits, same as memory or data packets traveling across the internet.

I also don't really see how the above couldn't be described as a shell script (ignoring static typing for now), something like

Code: Select all

# create new dri handle, something like /gpu/dri/inst0 or whatever.
# /gpu could be on a completely different computer for all we care
HANDLE=$(cat /gpu/dri/create)

# convert from jpeg to bitmap
jpegdecode smile.jpg > smile.bmp

# render bitmap to dri handle at [0, 0]
echo 0 0 smile.bmp > $HANDLE

seems pretty similar, unless there's some deeper semantics to the code snippet you posted that I'm not understanding.

The existence of shared libraries isn't a problem. Requiring a shared library in order to do IPC with a service, is not so much a problem as it is a sidestep that I am now hoping might be unnecessary. What I'd like to do is reference some function that a service provides as an unambiguous filesystem node, which should be a child node of the service's process folder. If the function is attached to a library instead of the service itself, it may go from “service:function()” to “service/lib:function()” which is far from a touch of death. I was planning to get around this with metadata, and maybe that's still a superior solution. Or the way metadata works simply needs to shift to accommodate it. Whatever.

I don't see why the function's handle would change if the function is provided by a shared library (I'm assuming we're still talking about 'traditional' libraries), the function is within the service's process, right?




Post scriptum of sorts:

testing the ideas against those requirements is how I evaluate what I'm working on.

Do you have an implementation of some sort people could look at? I'm having a hard time following your posts, and some concrete code could maybe help clarify things. At least from my perspective, it sounds like a lot of the stuff you're talking about could be implemented on top of existing kernels for testing. Wouldn't need to be complete or useful, but some kind of 'this is what I'm going for' would probably help.

[VincentVL85]

Yeah, I kinda figured after I posted that it might be getting away from the point, alas. I have a very bad habit of following a derailed train of thought, and I wrote that pretty early in the morning. This post has been edited to chop out an enormous amount of heavily depressed nonsense and rambling, but I'll go ahead and say no, I have no code. I wish I did, but then, I wish a lot of things were different. To bring it back around as much as possible...

Still, it's not obvious to me why the application's memory should be shared across computers at all. If the point is that different nodes implement different services, why can't I just send messages to these services and let each node handle its own memory?

Distributed applications have both distributed logic and distributed state. This chopped up app that I'm talking about is not a client-server model; it is not necessarily centralized at all. The entire app may be sleeping until any one of the fragments of the application detects a state change - a GUI event, an input device event, a network event, a clock event, a message - and starts an action chain, and the action chain may not start, end, or even pass through central processing. The goal as I said, is to claw back some performance lost to being a distributed/networked application, but doing that means preparing things so that if something does come up, any given app fragment has all that it needs to respond appropriately. It's an open question what the optimal way to arrange such a system is (unsolvable, probably), but I think it starts with accessing and possibly caching distributed state. That may mean making a request, it may mean keeping state synchronized, it may mean a lot of different things, and different solutions will be better or worse in different circumstances.

As far as not letting people do 'incorrect' things... it's my assertion that a system that doesn't leave room for absurd hacks will not succeed, because those absurd hacks are correct or at least surprisingly reasonable with uncomfortable regularity. Relatedly, it's not necessarily the case that a decentralized application needs fundamentally concurrent/parallel core logic. There may be one primary thread of execution that just so happens to jump around between machines when it reaches a stopping point, using synchronized state. Yes, race conditions and all that, it's difficult, I don't know.

Shared memory is one of those things I have thought entirely too much about given my inexperience and lack of education in the subject. I'd like to think I'm not like that with every single thing, but some subtopics I have given vastly more thought than I should. Shared memory touches on a lot of concepts at the heart of the project and so it feels like I should be touching it, but I may well have nothing meaningful to say, especially now. That said, a better understanding of IPC mechanisms would be very useful for me to keep my thoughts straight. Specifically, I have to admit, I am unsure how to correctly juggle host, client, user(s), and system ownership of data when a user-run app is making use of a service not necessarily run by the same user, on some remote hardware node that for all we know might be compromised or be hosting a compromised app, across a network that might be infested with maneating sharks. It could well be that I just assume something incorrect or am approaching it from the wrong angle - again - but it feels like a tricky thing one way or another.

I suspect we're talking of slightly different abstraction levels. From my perspective, a file provides the operations open (you get a new handle)/read (you get stuff out)/write (you put stuff in)/close (you ditch the handle), which is about as well-defined as can be I guess you're more interested in higher-level interfaces implemented via files...?

I misunderstood your point and was being a pretentious fop for no reason... but also yes, I am interested in higher-level interfaces for files, specifically where the file makes claims/guarantees about its contents, and those claims/guarantees can represented as endpoints that can be read from, written to, or executed as though they were memory. If you can do that, if you can divide a file into data fields and members, the low level read/write functions become the wrong way to handle that file, or rather, that filesystem node. Necessary for the underlying implementation perhaps, but it becomes the wrong level of abstraction if a filesystem node claims to hold, for example, a single number. Usable in a pinch, but the generic interface is not what you would use if you know the node's type.

As to "I don't see how 'everything is a file' would imply that operations on files should be well typed"... I was conflating a couple things. Sorry, I can see how that went off the rails; there's at least one side topic I'd have to walk you through to explain my logic there. That happens because I have been thinking about this for twenty years without having anyone at all to talk to; it's why I am, generally, all over the place. But I will say that your version of the shell script is problematic for a decentralized application, because everything has to go through central processing. A more decentralized version of the same might store interim values as filesystem nodes instead of shell variables, with notation indicating that the shell doesn't wait for those executions to finish.

This is, I apologize, once again off-topic.

If there is one thing to get back to, it's the question of generic solutions. It's fair to say that thread migration and message passing can be made equivalent, but they don't necessarily have the same side effects. Message passing, for example, necessarily requires the service logic to, first, return to the event handler loop, and second, get through all preceding messages. While you can have priority message queues and interrupt signals, I feel like thread migration as an IPC mechanism would better allow you to monitor and respond to the state of the service, or otherwise interact with something that is categorically "busy".

Also... let me see if I can explain this well. Probably not.

One of the features of MAD's OS design is collecting everything that claims to implement a given API interface, into a collection keyed to the interface ID. This features directly into why it's useful for apps to be the thing you link against, instead of a library. I wrote a meandering, complicated headache of a blog post a little while ago about letting libraries claim to be a implement an interface for a class of devices, and having a library configuration file that associates it with a device node, specifically so that you could point to a specific file in the filesystem as an implementation of an interface on top of a device. The goal is that single file - you can use a configuration file, but it feels like a hack.

If you can link to applications, you still have the same problem with "runnable but not running" apps, which have possible associations that are not yet nailed down, but once the service is running, everything falls into place.

That's why I consider it better generic solution to that specific problem. What I consider to be the "interesting" problem, is: what makes this a better or worse generic solution, to this or any other problem in distributed computing? That's the kind of conversation I hunger to have. Any time we can put together a really good general-case solution to problems that currently only have special-case solutions, that's powerful.

It's not necessarily something that needs feedback now, but I'd ask you to think about it if you find yourself idle. What does each solution assume? How likely are the assumptions to be incorrect? If your assumption is incorrect, how do you detect that, what are the consequences, and what is the fallback? If it doesn't seem like your kind of thing, by all means don't let it distract you. But it is where I am coming from.