Migrating threads as IPC

[AndrewAPrice]

I went through several iterations of this over the past 2 decades. My guiding principal is that IPC should (at least from the userland perspective) be as simple as calling a function.

A very long time ago, during they heyday of .Net, I had this idea that by requiring "safe" languages (basically all executable code is byte code, and the kernel is an interpreter or JIT compiler), you could disable hardware memory protection (identity map all memory), and loading programs simply ensured their code was in memory and you started executing a function, and there would be no "system calls" since you would just issue function calls, and any global or static variables would be 'system wide', and some "trusted"/"unsafe" methods would be allowed to do IO. If code could be garbage collected, then terminating a thread would allow us to clean up unreachable code from memory.

This was great in theory and I did write a functioning compiler and bytecode interpreter, but it became daunting since I'd basically have to reinvent the wheel and could never port software to my system. Maybe native code and supporting C/C++, etc wasn't so bad? So I moved to a traditional kernel with paging.

I really liked the idea of "Function calls as IPC". So two ideas emerged out of this:
1) Userland programs register entry points with the kernel, and RPCs froze the caller and the kernel spawned a thread in the userland.
2) IPC was zero-copy, so there would be no overhead if you send 1 byte or 1 MB.

So in practice, the "kernel spawning threads" turned out to be difficult. C and C++ has thread local storage with constructors that run before the entry point. It's up to the kernel to handle stack management, to assume ABI, etc. So I abandoned that to more traditional messaging - "hey there's a message", but my intensive investigation into this stumbled upon fibers. They're similar to threads, but they're managed in user land and cooperatively multitasked - that is, instead of being interrupted, they run until they're blocked (e.g. stuck at a mutex, sleep for a duration, wait for an RPC), and instead of sleeping the thread running the fiber stack, you jump to the next fiber. And because context switching only happens as part of a function call, you only need to switch call stacks and preserve the callee-preserved registers. And so now, each userland process has a message loop, that when something comes in (RPC, a timer expired, a futex unlocked) we spawn or awaken a fiber, and it's very cheap because I use an object pool of recycled stacks and it's all implemented in userland.

Now for zero-copy IPC, I was went down another rabbit hole. I invented an IDL called Permebufs, which were data types defined in code, I use a code generator to generate C++ stubs for these types. This worked, although the API was clunky. The code looked somewhat like:

Code: Select all

Permebuf<CustomRequestType> request;
request->SetSomeValue(1234);

StatusOr<Permebuf<CustomResponseType>> response = myService->CallSomeFunction(std::move(request));

`Permebuf<T>` was the outer wrapper, which was some page aligned memory that could grow and only contained that an instance of `T` as the root object, and you'd write directly into those memory pages. Then to make an RPC, the kernel would unassign memory from the caller and 'gift' that memory page to the client.

I had a working implementation but it was clunky.

• It was clunky API. It was a write-once data structure, copying data from one Permebuf<> instance to another had to do a deep copy.
• It was clunky in memory, because even though I tried to optimize the most common case by letting userland programs keep a pool of unused memory pages, sending between programs still required the kernel to scan their address space to find an unused range to map it into.
• At the time, I used a Javascript-based build system, and the Permebuf code generator was complicated to maintain and embedded into my build system, and I wanted to switch over to better build system, and I didn't want to rewrite it.

I eventually gave up on Permebufs and 'zero copy IPC' because the vast majority of messages are super small anyway, and I had shared memory, so anything large (like reading from a file, creating a UI texture), the RPC request would contain the shared memory ID and an offset or something. Also, I wanted to get rid of the custom IDL and code generator, so I went with a macro system build a light-weight serialization library.

Now, to define a service:

Code: Select all

#define METHOD_LIST(X)                                                         \
  X(1, OpenFile, OpenFileResponse, RequestWithFilePath)                        \
  X(2, OpenMemoryMappedFile, OpenMemoryMappedFileResponse,                     \
    RequestWithFilePath)                                                       \
  X(3, ReadDirectory, ReadDirectoryResponse, ReadDirectoryRequest)             \
  X(4, CheckPermissions, CheckPermissionsResponse, RequestWithFilePath) \
  X(5, GetFileStatistics, FileStatistics, RequestWithFilePath)

DEFINE_PERCEPTION_SERVICE(StorageManager, "perception.StorageManager",
                          METHOD_LIST)

#undef METHOD_LIST

And the types can be regular C or C++ structs/classes:

Code: Select all

class RequestWithFilePath : public serialization::Serializable {
 public:
  RequestWithFilePath() {}
  RequestWithFilePath(std::string_view path) : path(path) {}
  std::string path;

  virtual void Serialize(serialization::Serializer& serializer) override  {
    serializer.String("Path", path);
  }
};

class FileStatistics : public serialization::Serializable {
 public:
  bool exists = false;
  DirectoryEntry::Type type = DirectoryEntry::Type::FILE;
  uint64 size_in_bytes = 0;
  uint64 optimal_operation_size = 0;

  virtual void Serialize(serialization::Serializer& serializer) override {
    serializer.Integer("Exists", exists);
    serializer.Integer("Type", type);
    serializer.Integer("Size in bytes", size_in_bytes);
    serializer.Integer("Optimal operation size", optimal_operation_size);
  }
};

(The string is only used for pretty-printing, but the order of the operations matter - the `serializer` knows the expect int, int, int, int.)

So, while under the hood they're not true function calls, I got as close as possible API-wise. This is how you issue an RPC:

Code: Select all

auto status_or_response = GetService<StorageManager>().GetFileStatistics({pathname});

And the implementation looks like:

Code: Select all

class StorageManagerImpl : public StorageManager::Server {
virtual StatusOr<FileStatistics> GetFileStatistics(
    const RequestWithFilePath& request, ProcessId caller) override {
   // ...
}

And so.. while this got off topic, this has been my journey. I'm still interested in software isolation and sharing threads across process boundaries, but in practice with making a functional operating system, I was only able to achieve this in the API.