Building a Decentralized Operating System

[Josephjoe]

Hey guys, So I've been working on a new protocol called the Marketplace which is a decentralized operating system that co-ordinates and economizes the execution of computational work across a peer-to-peer network of nodes. Where there is no barrier to the node participation.

Unlike proof-of-work systems, where nodes burn large amounts of energy to solve "non-useful" puzzles, the Marketplace organizes a peer-to-peer market of computational trade where nodes offload useful computational work called "jobs" directly to each other and pays in the system's native cryptocurrency, goldcoin(GDC). Effectively redirecting energy into real economic growth.

Security without "Staking" is achieved using Proof-of-Capability (PoC), a new "sybil-resistant" mechanism that selects and incentivizes a small committee (“whiterooms”) to validate and reach consensus on the result of jobs without boggling down the entire network with redundant execution. This allows the amount of jobs handled in parallel to scale directly with the amount of nodes on the network analogous to an OS on a multi-core device.

Real utility then comes from the "services layer" where nodes can compose stalls(modular services) into larger digital structures(e.g websites), and execute them regardless of size in near constant time by taking advantage of the parallel execution environment of the marketplace. The system’s monetary policy dynamically adjusts issuance such that price of execution is constant regardless of network load.

Whitepaper (PDF):
https://github.com/bajoescience/Marketp ... epaper.pdf

I’d appreciate feedback on the design, especially on consensus security, speed and

the economic model, Thanks.

[Octocontrabass]

protocol called the Marketplace which is a decentralized operating system

I'm not sure a protocol counts as an operating system. The distributed computing aspect is interesting, though.

cryptocurrency

Does anyone actually use cryptocurrency as currency, or is it all get-rich-quick grifters trying to make the line go up so they can turn a profit before the line goes back down?

"sybil-resistant" mechanism that selects and incentivizes a small committee

So an attacker only needs to find a weakness in the committee selection process to take control over the network.

compose stalls(modular services) into larger digital structures(e.g websites)

What kind of website are you imagining that's simultaneously big enough to require distributed computing and small enough that the operator can't buy computing resources with normal money?

[Josephjoe]

I'm not sure a protocol counts as an operating system. The distributed computing aspect is interesting, though.

Yes, it's like a scheduler that distributes work onto idle nodes around the network, it's basically your average OS, but instead of multi-cores, you have a network of nodes. But's it's even simpler than an OS, The system does not choose the node("core") to offload some work to, the node needing to offload work can easily find the node that is idle, the Marketplace simply makes sure consensus is reached on the correctness of the job and the payment thereafter. This is why I called it a protocol.

It is also supposed to reside in kernel space and be as close to hardware as possible. Although it's only core protocol that resides there, the main interface that actually does work is the RISC-V based VM.

Does anyone actually use cryptocurrency as currency, or is it all get-rich-quick grifters trying to make the line go up so they can turn a profit before the line goes back down?

If you notice, this is called "cryptocurrency" simply because it has cryptography primitives in it. This currency is tied to real energy because nodes spend energy to do CPU work, and then are paid in the currency. This makes it inherently valuable regardless of fiat exchanges. (I hate fiat though)

So an attacker only needs to find a weakness in the committee selection process to take control over the network.

No, because of the PoC mechanism I talked about. This mechanism is like a valve before the committee. It works in such a way that it's easy to guarantee one seat in the committee. but the more seats you try to guarantee, the exponentially harder it becomes.(Not that it's truly impossible) This is without staking or wasting resources like POW, so like a hybrid version. I personally think this is the most innovative part of the whitepaper.

Even if somehow they are able to take control of a whiteroom which is max 100 seats, they can only falsify one job in a pool of millions-to-billions of jobs, now to take control of the network, I can't forsee that!

What kind of website are you imagining that's simultaneously big enough to require distributed computing and small enough that the operator can't buy computing resources with normal money?

Okay good question, From the earlier answers, I have tried to paint a picture of a system analogous to a device with potentially millions of cores. This allows for programs to truly be run in parallel. If we can break down the services that make up a website into chunks, break down those chunks into smaller chunks to the point where they can exist as the smallest piece of independent logic(e.g function in the sidebar), theoretically, we can run all these in parallel by combining them into an app workflow on the Marketplace, the workflow triggers branches asynchronously(reference Whitepaper)such that In these scenarios, No matter the overall composed size of the App, the speed of the app is the speed of it's slowest component.

Also, the marketplace has a fixed price for compute as I said, which will always remain very cheap(in fiat) because of the Tokenomics behind monetary inflation in the system, and the consequence of the decentralized market.

Whitepaper: https://github.com/bajoescience/Marketp ... epaper.pdf

[Josephjoe]

So I posted a similar thread on reddit (https://www.reddit.com/r/CryptoCurrency ... t/nbc33xc/), and a redditor (u/Less_Result5615) just suggested a very good use-case, What do you guys think :

I read your whitepaper, and I loved your ideas. Here are my thoughts.

The Marketplace system could be used to optimize massive open source codebases like Linux. A sponsoring entity, such as a foundation or a DAO, would run a Layer 2 orchestrator application. This application would parse the entire Linux source code, identify thousands of individual functions, and automatically create a distinct "job" for each one. Each job would package the function's source code, a suite of unit tests for correctness, a performance benchmark to measure speed, the current performance score to beat, and a GDC reward. The orchestrator would then broadcast these thousands of jobs to the network simultaneously.

As the network receives these jobs, the system's parallel architecture would take over. For each individual job, a small, independent Whiteroom committee would be formed to handle the optimization and validation. This means thousands of different functions from the Linux kernel would be worked on in parallel by different, competing groups of nodes across the globe. When a worker node finds a potential optimization, it submits the new code. The other nodes in that Whiteroom then deterministically verify the submission by compiling it, running the unit tests, and confirming that it's faster via the benchmark.

Once a Whiteroom reaches consensus on a valid, superior version, the job is finalized. The orchestrator application receives the verified, optimized code snippet. This app could then be programmed to automatically create a pull request on the official Linux repository for review by human kernel maintainers. The pull request would include not just the new code, but also the verifiable proof of its improved performance. This process transforms code optimization from a slow, manual task into a continuous, economically incentivized, and massively parallel global effort.

In theory, this could work on a any large cumbersome open source codebase: Linux, FreeBSD, MariaDB etc

[Octocontrabass]

It is also supposed to reside in kernel space and be as close to hardware as possible. Although it's only core protocol that resides there, the main interface that actually does work is the RISC-V based VM.

Why does any of it need to be in kernel space? It's a network protocol and a virtual machine, neither of those need kernel privileges. In fact, you probably don't want them to have kernel privileges because any bugs in your implementation can turn into huge security problems.

If you notice, this is called "cryptocurrency" simply because it has cryptography primitives in it. This currency is tied to real energy because nodes spend energy to do CPU work, and then are paid in the currency. This makes it inherently valuable regardless of fiat exchanges. (I hate fiat though)

But it can be exchanged outside of buying and selling CPU work, right? That makes it a cryptocurrency.

No, because of the PoC mechanism I talked about.

Sure, but the committee selection occurs on only one node, right? So the attacker only needs to prevent that node from receiving legitimate attempts to join the committee to compromise the selection process.

Even if somehow they are able to take control of a whiteroom which is max 100 seats, they can only falsify one job in a pool of millions-to-billions of jobs, now to take control of the network, I can't forsee that!

If an attacker has figured out a way to falsify the results for one node specifically, from that node's perspective, they have taken over the entire network.

a redditor (u/Less_Result5615) just suggested a very good use-case, What do you guys think :

Where are those unit tests coming from? Are they automatically generated too? If you can automatically generate correct unit tests for existing code, go sell your algorithm to Microsoft. (LLM-generated unit tests are wrong too often to be useful.)

[Josephjoe]

Thanks for your reply, let me address your concerns.

Why does any of it need to be in kernel space? It's a network protocol and a virtual machine, neither of those need kernel privileges. In fact, you probably don't want them to have kernel privileges because any bugs in your implementation can turn into huge security problems.

It is only the minimal core protocol (essentially the job scheduler + consensus dispatcher) that resides in the kernel, just enough for jobs to be treated as syscalls. Everything else is in user space including VM. This is due to efficiency and security, also overall management of active jobs in a node which is why it's considered a part of the protocol. Either way

But it can be exchanged outside of buying and selling CPU work, right? That makes it a cryptocurrency.

Yes you are correct, but the currency isn't speculative by design as it's pegged to demanded work / energy. nodes pay directly to each other for cpu cycles spent.

Sure, but the committee selection occurs on only one node, right? So the attacker only needs to prevent that node from receiving legitimate attempts to join the committee to compromise the selection process.

Thew whiteroom committee is not a global co-ordinator, but it's built for that node's job ONLY. This means that the committee acts as a trusted consensus voice only on matters concerning that job, and it is the employer that assembles the whiteroom committee blindly. The whitepaper goes into detail on how this process is secure.

If an attacker has figured out a way to falsify the results for one node specifically, from that node's perspective, they have taken over the entire network.

NO. The whiteroom is local to job, this means for an attacker to control the network, they would have to control a majority of whiterooms across the network which is economically infeasible.

Where are those unit tests coming from? Are they automatically generated too? If you can automatically generate correct unit tests for existing code, go sell your algorithm to Microsoft. (LLM-generated unit tests are wrong too often to be useful.)

Fair point, but the person spoke of not generating tests (human written), but to create optimizations that produce faster benchmarks for each part of the codebase. But the true value is that it can run very large test suites in parallel with deterministic verification, making it practical to validate huge codebases at scale. That’s the novelty

Whitepaper: https://github.com/bajoescience/Marketp ... epaper.pdf

[Octocontrabass]

It is only the minimal core protocol (essentially the job scheduler + consensus dispatcher) that resides in the kernel, just enough for jobs to be treated as syscalls.

That doesn't really answer my question, though. Why do jobs need to be treated as syscalls?

the currency isn't speculative by design as it's pegged to demanded work / energy.

What stops your currency from becoming unpegged from its stable asset the same way other "stablecoins" have?

Thew whiteroom committee is not a global co-ordinator, but it's built for that node's job ONLY. This means that the committee acts as a trusted consensus voice only on matters concerning that job, and it is the employer that assembles the whiteroom committee blindly.

In other words, an attacker only needs to compromise the employer's node in order to compromise the employer's jobs.

The whiteroom is local to job, this means for an attacker to control the network, they would have to control a majority of whiterooms across the network which is economically infeasible.

It's only economically infeasible if the network grows large enough to make it economically infeasible. But I'm not even talking about taking over the entire network, I'm talking about taking over one specific node to compromise one employer's jobs.

But the true value is that it can run very large test suites in parallel with deterministic verification, making it practical to validate huge codebases at scale.

That's already possible with today's cloud computing.

[Josephjoe]

Thank you so much for your patience. This time I'll try and be as extensive and direct as possible.

That doesn't really answer my question, though. Why do jobs need to be treated as syscalls?

First, The VM is RISC-V based, but it only runs in U-mode. This allows the VM to safely run un-trusted stall code(deterministic services). These stalls are potentially not a single monolith, but can be divided into smaller components which are themselves stalls. These components can be executed in parallel, and then merged for a final result.

The encompassing stall then triggers an ecall asynchronously to have it's sub-components run in parallel, without blocking it's own execution.

In a mulit-core device, the OS would be in charge of this process by handling the syscall and distributing the parallel work in the most optimized way across it's cores. This process has a limit because there is ultimately a finite amount of cores.

The Marketplace protocol handles these ecalls by distributing the parallel work amongst nodes/cores on the network. Well not actually "distributing" because that will make it an OS. But it simply organizes and matches worker cores with employer cores in an economized and secure way. The difference I highlight here is that the OS decides by itself which core executes which component, while the protocol allows the system to find another core to offload work to in exchange for money. (Each core becomes an economic entity)

Since it's not a full OS, it can exist as a daemon of an OS, just built for specifically this use-case without thinking of other OS functionalities. Also, jobs as syscalls allows stalls to be composable and be executed asynchronously. without this there would be no use for the Marketplace.

What stops your currency from becoming unpegged from its stable asset the same way other "stablecoins" have?

That's a good one, But ultimately "pegging" in this sense takes trust especially when connected to fiat and centralized issuers (stablecoins). But here, “peg” is enforced natively by the protocol: jobs always cost the same amount of goldcoin per CPU-second (adjusted dynamically by issuance). If demand spikes, more GDC is minted; if demand falls, GDC burns through fees (all happening in a decentralized way).

In theory, that keeps execution cost constant in GDC terms, regardless of speculation outside the system. Of course, if adoption is low, volatility still leaks in, so it’s more of a “protocol peg” than a fiat one. But given there is demand in the system which is provided by the stall layer, the goldcoin value will be directly connected to CPU cycle costs.

In other words, an attacker only needs to compromise the employer's node in order to compromise the employer's jobs.

Yes, but the main question becomes "At What Cost?". That's where the POC comes in as it uses a small VDF(Verified Delay function) to ensure that it becomes exponentially harder to guarantee more seats in the Whiteroom committee for which >2/3 of all seats is needed to control.

Also, if an employer is compromised, it's jobs are unsafe just like a compromised client in any distributed system. The important thing is that this doesn’t cascade into a global takeover (other employers and whiterooms remain unaffected.)

That's already possible with today's cloud computing.

True, But the marketplace is decentralized and peer-to-peer. Instead of a centralized outpost, it creates a market where millions of people can take part and be economically incentivized. A re-distribution of wealth. That does not mean that centralized cloud is being replaced, given that the marketplace has limits (see Whitepaper). But instead cloud computing is given a more niche role. Both can also be combined at the App layer of the marketplace.

[Octocontrabass]

The encompassing stall then triggers an ecall asynchronously to have it's sub-components run in parallel, without blocking it's own execution.

Okay, so jobs are only syscalls from the perspective of the guest code? That means the protocol can be implemented entirely in user mode and doesn't need any dedicated syscalls on the host OS, right?

Also, if an employer is compromised, it's jobs are unsafe just like a compromised client in any distributed system.

Sure, but my concern is more about how easy it is to hide the fact that the employer is compromised. If it's possible to manipulate the whiteroom just by delaying network traffic from nodes that don't belong to the attacker, the attacker doesn't even need to compromise the employer's node - just the upstream network devices responsible for connecting that node to the rest of the network.

A re-distribution of wealth.

I'll believe it when I see it.

[Josephjoe]

Thanks for the reply, so let me go over it:

Okay, so jobs are only syscalls from the perspective of the guest code? That means the protocol can be implemented entirely in user mode and doesn't need any dedicated syscalls on the host OS, right?

The scheduling important here, jobs coming from the network are grouped into priority classes, some jobs are ignored or executed according to the host's workloads to avoid starvation. The protocol tags Marketplace tasks so the scheduler can preempt, cap, and account them cleanly.

The protocol is simply the part of the OS that allows jobs of this kind to be scheduled and accounted for accurately (needed for cpu cycles gauge)

Sure, but my concern is more about how easy it is to hide the fact that the employer is compromised. If it's possible to manipulate the whiteroom just by delaying network traffic from nodes that don't belong to the attacker, the attacker doesn't even need to compromise the employer's node - just the upstream network devices responsible for connecting that node to the rest of the network.

The marketplace relies on gossiping and multi-peer networks. not perfect, but can be improved. But even then there is only so much protocol can do against network level attacks concerning ISP. Although I'm looking into mesh networks for that.

I'll believe it when I see it.

Agreeable!! Let's build it and put it to the test.
So there are two currencies in this project, goldcoin(GDC), and Gold Trust Token (GTT). The GTT represents the shares of all market fees collected in goldcoin on all jobs across the network where the fee is 0.001% of job work price. There is a private company that distributes the GTT according to a schedule as described by the whitepaper, and gets all the profits from that which is the company I own.

I'm offering 20% equity of the company that distributes the GTT to you to help me build a mvp(for at least developers). Given that I do not yet have the tech skills to put this together.
We build it in rust according to the whitepaper and with an additional dev, maybe even potential changes given enough debate. What do you think?

[Josephjoe]

The encompassing stall then triggers an ecall asynchronously to have it's sub-components run in parallel, without blocking it's own execution.

Okay, so jobs are only syscalls from the perspective of the guest code? That means the protocol can be implemented entirely in user mode and doesn't need any dedicated syscalls on the host OS, right?

Also, if an employer is compromised, it's jobs are unsafe just like a compromised client in any distributed system.

Sure, but my concern is more about how easy it is to hide the fact that the employer is compromised. If it's possible to manipulate the whiteroom just by delaying network traffic from nodes that don't belong to the attacker, the attacker doesn't even need to compromise the employer's node - just the upstream network devices responsible for connecting that node to the rest of the network.

A re-distribution of wealth.

I'll believe it when I see it.

So overall, The marketplace extends OS logic from among cores in a device to a network of nodes. therefore turns those network nodes to a multi-core system. But it's also more like a specialized OS service, and this means it can be seen as an extension of the OS.

Processes(stalls) are scheduled across peers on the network, with consensus ensuring correctness and tokens ensuring fairness.

[Josephjoe]

Okay, so jobs are only syscalls from the perspective of the guest code? That means the protocol can be implemented entirely in user mode and doesn't need any dedicated syscalls on the host OS, right?

Sorry, You are absolutely right here on fourty-fifth thought. Given that the core protocol simply behaves like a scheduler, It can be made initially as a user-level process. This can then be implemented as a main manager process that selects jobs to be done, handles mempool storage, and triggers the real OS to spawn VMs to handle those jobs.

I really misunderstood you earlier, as ecalls are from the perspective of the stall source code, not the VM itself. But either way, for maximum efficiency, I still intend to make a Operating System version later when this project has really taken off. But this one would be a complete monolithic OS with all the components, and device drivers needed. I'll probably call it MarketOS similar to Linux. So yeah for now it's a user-level OS.

Thanks Joseph.