(MAD) Distributed Applications Model

[VincentVL85]

So I introduced myself in a post previously as being the sole representative of some "Project MAD", and then I've been silent here, mostly on account of me being extremely depressed and it being very difficult to motivate myself to explain even those things that are fairly obvious, much less things I know will be confusing to people. During that time, at least, I've been able to post more on my blog, linked in the signature, but that's dense and confusing and I don't blame anyone who chooses not to read it.

Instead I'd like to introduce a concept here, of a distributed applications model, one of which is a component of Project MAD. I'm sure many of you will question the why, and while I try to address it at the end, this post is somewhat deeper.

The general idea of a distributed application is that it is multiple application fragments, designed and written together, which may be deployed onto either the same physical machine, or multiple different machines connected by a network. In the process of deploying the fragments, network connections are established and maintained between fragments. Whenever the code reaches a point where one fragment contacts another, the back end (a library or service) routes the call either across the network, or to the local function as appropriate.

I introduced this as a model, but it can only be called that when certain questions are answered. The most important for the moment, being how the application is structured and divided to make the above both functional and useful. My model, which I have been calling the Agentic Distributed Applications model (ADA), is my answer to that.

Structurally, I envision something like a C program with namespaces, where each namespace becomes a separately deployable fragment. Each namespace has one or more data egress and ingress points; if the data passes over the network, these are used to pack, unpack, and verify data. These points are likely to be machine generated or standard; the point is for them to abstract away the question of whether there is a network in between the fragments, and while a programmer might extend their functionality, the basic functionality is very generic. Important to understand here, the compiler can (should, shall) automatically turn cross-fragment calls into egress point calls - meaning the code as written is simply a call to the correct function (or a read from/write to appropriate memory), with the middleware abstracted away.

More interesting to me is the question of how a program is divided into fragments. My answer is fundamentally a mater of consuming external capabilities; if your program relies on a device or service, or can be considered an independent (or workflow-able) compute task, then it is to be separated. Input is separate; output is separate; compute is separate; there may be event handlers in each fragment as a matter of necessity, but again, these should be generic and ideally machine-generated (unless manually extended or replaced).

If this is made to work, the point is an application that works identically when, for example, you are using a computer as a dumb terminal and running the guts of the app on another machine, or using one machine as input and CPU and another machine as A/V out. But - and this is the crux of it - the model is generic. It works just as well when you are talking about things like controlling IOT and PAN devices; it works when you are running an app on your phone and displaying it in your car or on a TV. It works when you are running a game on a server (home or otherwise) and playing it in your living room.

Whether my model works or not, the idea of applications that can abstract away the boundary between two machines is, I believe, important. If I fail at everything else, that's an idea I'd like to spread.

However. This is all, so far, my personal thoughts with no input from anyone. In general I try to anticipate problem points and consider how they may or may not be addressed, but both my theoretical and practical experiences are limited. So I need to ask:

What do you think you see in my description? What sounds right? What sounds wrong? What problems do you think are lurking? Where have I completely failed to explain what I'm talking about? I honestly don't know how I come across. I don't exactly have a lot of peers. This project... is a little mad.

Thoughts? And thanks.

[Octocontrabass]

If this is made to work, the point is an application that works identically when, for example, you are using a computer as a dumb terminal and running the guts of the app on another machine, or using one machine as input and CPU and another machine as A/V out. But - and this is the crux of it - the model is generic. It works just as well when you are talking about things like controlling IOT and PAN devices; it works when you are running an app on your phone and displaying it in your car or on a TV. It works when you are running a game on a server (home or otherwise) and playing it in your living room.

So, I get where you're coming from, but these examples are all things you can do with existing applications on existing operating systems, just by redirecting input and output across a network. The application itself wouldn't need to be fragmented.

What do you think you see in my description?

The most compelling part, to me, is the possibility that your application model makes it easier to develop distributed applications.

[VincentVL85]

So, I get where you're coming from, but these examples are all things you can do with existing applications on existing operating systems, just by redirecting input and output across a network. The application itself wouldn't need to be fragmented.

This is exactly the point. This is intended to be a general-case solution. There are existing speical-case solutions, and there are attempts to have a narrower general-case solution. But there can't be a broader solution without understanding the general case.

What interests me, and what I am hoping to encourage others to look into, is a model for solving all or nearly all cases of the same problem at once. That's possible, if you understand the general goal and constraints. As long as there is good compute on both sides (this is harder when you start looking at microcontrollers, IOT and PAN stuff) and reasonable networking, you CAN abstract away the distance between. The only question is what that solution looks like.

Once you know that you are trying to develop a pattern that works in this general case, I feel like it makes sense. You still will want to create accessory standards and services, but I feel like a general-case pattern is valuable.

[Octocontrabass]

This is exactly the point. This is intended to be a general-case solution. There are existing speical-case solutions, and there are attempts to have a narrower general-case solution. But there can't be a broader solution without understanding the general case.

But your examples already share the same general-case solution: IPC over a network. Plan 9 could do that 30 years ago, so unless I've missed something, this is nothing new.

The new part, at least to me, is streamlining development of application fragments that communicate with each other using IPC.

[VincentVL85]

Ah, Plan 9. I was so excited to find out it existed, but while it was, perhaps, at one time, arguably complete, it wasn't and isn't ready.

There are... a lot of arguments I could make, a lot of angles to approach. It's part of the reason why I have a blog that rambles on endlessly and is hard to parse. Here's one: to paraphrase this blog post... something being available is not the same thing as having presence on that hardware. Presence is important for debugging, network issues, cleanup... a lot of things.

I'm not just trying to shut you down. I laid the groundwork for the design of MAD 20 years ago. When I started, I thought that the whole thing would run over RPC calls, and I had a nagging feeling in the back of my mind that told me it didn't make sense, like I was missing something key. I wasn't focusing on this project for those 20 years, but I did keep revisiting it, and I kept feeling like I couldn't make it work the way I had it in my mind. When I tried to organize my thoughts, they'd grind to a halt.

Some pieces of MAD that are quite frankly more mad than this model, were born to try to make RPCs work as the core of a model - like integrating object type data into the filesystem. I'll still do that if I can, but it felt truly necessary when the core concept was your program sitting in one place and throwing messages around to every device and service scattered across the system (using the filesystem as a directory, if that isn't clear from context). Because a distributed application model, fundamentally, has all the network traffic being between your program and itself, instead of your program and other programs, that kind of type safety worry is no longer critical; important, yes, but not critical. Instead of having to solve a lot of problems at the network and/or filesystem level, you can use the same programming techniques we have now, libraries and such, and they just so happen to not all be on the same machine.

If you feel like the existing technology could work, I invite you to think a few steps ahead, and I don't mean that cynically - again, part of the point here is thinking and feedback. Think about programming a GUI app where the display server, input, storage, and compute are all on separate machines. If you think that all it takes are P9-style exposed functions, then try to understand how that model actually works. Try to pseudo-code it out. I suspect you'll hit a point where you hesitate, and a concept that makes intuitive sense is harder to actually implement than it seemed from a distance.

I'm not saying you won't find a solution. But I went mad trying to make the model work with RPC, and people walked away from Plan 9. We need something, and I don't think we have it yet.

[Octocontrabass]

Think about programming a GUI app where the display server, input, storage, and compute are all on separate machines.

How is that different from programming any normal GUI app? (Or is your point that it isn't any different?) Your program runs on the machine with compute resources and asks the OS to provide display, input, and storage resources. Your program doesn't need to know or care how the OS provides those resources.

[VincentVL85]

I guess when I say GUI I really mean direct rendering apps, despite knowing those aren't the same thing at all - the point being here that rendering updates crossing the network instills both a time crunch (frame rate) and network congestion. And since multiple GUI applications may be competing to get in before the refresh, they will all be competing for space in the same general window. It's my assertion that you want to touch the network as little as possible, for both latency and congestion reasons, whenever dealing with these things. Now, GUI apps that don't have a custom render pipeline that has to cross the network, those don't get hit so hard; they're fine with relatively simple and direct links. If that's what I implied, my bad.

Note that one of the complications I threw in my scenario is that the data storage is separate from both compute and the display server. I didn't, but could, also add that the display device is somewhere separate from the display server (eg USB). Even without that, we cannot necessarily assume that the display server's ram is adequate to hold every bit of visual data that might be needed at once, so it may need to refer back to the disk or the program memory. In principle, there are a lot of tasks here that should be going on in parallel, and many of them are specific to your application, not generic tasks. Easy, generic tasks, those you might cue without having presence on a given machine, but if you need your own code to run, then you need to manage that compute task, whether it's centrally or distributed.

It may be possible to create an optimized workflow with a centralized application, where you cue things you will need shortly, have them arrive on time, and then send them elsewhere - assuming you know what you need ahead of time, of course. You can use multiple threads on the compute core to handle all the parallel workflows and get things as smooth as possible. But if for example the display server has invalidated some memory block and it needs to be fetched from disk, that bounces through the program center, and because you don't have presence on that machine, you might not discover that this is necessary until you make a request and get an error (that would be a poorly designed system, I admit, but I'm thinking worst case). Distributed applications would not only let you have presence to receive and parse errors and events local to the event itself, but enables many-to-many messaging so that events do not need to bounce through a central compute node (who will probably be processing these messages in a single central queue).

Like I said, there are a lot of angles I could approach the question from. If you sit there and think of the worst things that could possibly happen, and try to figure out how you would deal with it on an architectural level... well, I can tell you it's a frigging headache, lol.

[Octocontrabass]

Ah, so your point is that those scenarios will work better under your design, not that those scenarios are only possible under your design. I think I understand where you're coming from now.

But it's a lot harder to prove that your way is the better way of doing things if you haven't actually implemented it. That's especially true when your claims are changes in performance like lower latency. How can you know that your method offers lower latency without measuring it?

[VincentVL85]

Believe me, I'm well aware of the practical problems that come with me not being able to just do this myself. I have a BSCS but never took OS dev courses, and my brain just doesn't like to do complex code work. I have a lot I want to do and I've tried, but it just makes my brain cramp. Quite frankly I don't know anyone, personally, who does OS dev work, and I'm not great at making friends. Can't imagine what any of that has to do with me posting here.

To the rest of your point,

I would more peg my point this way: as technology becomes more mature, we need to standardize behind some model for what we're doing. Every time we take a difficult problem and just provide a standard that solves it, a lot of people jump on it. Before that, even if we have the technology, you see a lot of people standing around looking confused and a few people trying things experimentally, but few if any great motions towards anything.

Thus my point is, if you think existing technology, P9/9P and so on, are worth forming into a standard, then do the due diligence on it - not formally, but just, in your mind. If I had the knowledge and ability to make a presentation, a request for comment, on a wider scale, I would. It'd be embarrassing how many holes in my design I have, but I'd still do it, because I believe the core idea is important and needs attention. The piece that I think works, the piece I believe in, is the high level organization of a system, based on distributed applications. And quite frankly, I'm having enough trouble explaining everything, let alone formalizing it.

The best I can do right now is try to explain one piece at a time and see if anyone has any objections.

[VincentVL85]

So I am in the process of writing a blog post, about a topic related to this, and I ended up in kind of the same place as @Octocontrabass's objection (kind of but not really on purpose), specifically, if we can solve the problem with existing technology, what does the model actually do? And then in the attempt to write this reply, I reframed my thinking and erased the old reply for this one.

I'm not sure that I made it clear that part of the underlying assumption in my distributed application model is an OS designed around the idea of what I call "CPU for hire", aka serverless/cloud/lambda computing, in which you send code to a machine and it just runs, without installation; consider it a kernel-level EC2 model if you like, though my focus wasn't on spinning up more instances to handle many copies of the same workload (and actual payments are not involved or not the point). Instead I'm interested in designing the foundation of a personal, desktop computer under this kind of circumstance. (I tend to forget about EC2 and other cloud services because, being broke, I could never pay to use them, and also I don't have any tasks that would make good use of them. As such, I also don't really understand them, except generally)

In that context, part of the point of the ADA or any distributed applications model is: what does an application that lives natively inside this space look like? You can imagine many designs that don't deserve to be called "native" applications - apps that do almost everything in one place except where the developer manually adds a mechanism to do some random thing in the cloud, or whatever. Or apps that don't care about the network in between and try to bit bash a low-level device driver from a remote CPU process, instead of having that code be, you know, local.

Arguably, a native application in this environment would do as little crossing the network as possible. The logic of the application would send messages as necessary, keeping some distributed state and so on, but you don't send multiple messages when one will do. The keyboard for example won't send every message across the network for processing, and instead it will understand the application state and screen the inputs, sending only relevant ones to the application. And you wouldn't have any of your graphics rendering stack away from the GPU/windowing system unless you really need to.

Does that make any more sense?