HIC – A hierarchical isolation kernel with physical sandboxing and sub‑5ns capability checks

[DslsDZC]

I've been working on an experimental kernel called HIC (Hierarchical Isolation Core) and would love to get feedback from this community.

**What is HIC?**
HIC explores a three‑tier privilege model:
- Core‑0: a tiny (~10K LOC) trusted component that manages capabilities and physical resources.
- Privileged‑1: system services run in *physically isolated sandboxes* – they execute at the same CPU privilege level as Core‑0 but have independent memory regions enforced by the MMU.
- Application‑3: normal user processes.

**Key design points**
- *Physical space direct mapping*: Each Privileged‑1 service gets a contiguous physical memory region. This avoids TLB flushes when calling services and eliminates virtual memory overhead for those services.
- *Ultra‑fast capability system*: Capabilities are 64‑bit tokens (16‑bit domain ID + 48‑bit capability ID) validated in ~13 instructions (<5ns at 3GHz, based on instruction counting). The fast path is inlined.
- *Rolling upgrades with zero downtime*: Services can be updated without disconnecting clients. The Module Manager coordinates state migration, connection handover, and automatic rollback (design docs available).
- *Portability*: CHAL/AAL/IMAL layers abstract hardware differences – currently targeting x86‑64, ARMv8‑A, RISC‑V, and no‑MMU variants.

**Current status**
- **Custom bootloader**: Supports UEFI on x86_64 (hardware initialization, kernel image loading/verification, boot info passing). Designed with portability in mind (ARM, RISC‑V planned).
- **Core‑0 kernel**: Can boot on x86_64 QEMU via the custom UEFI bootloader, but the kernel is **very unstable and frequently crashes** – it's extremely early days.
- **Module Manager & rolling updates**: The code exists but has **not been successfully tested** – module loading and rolling update flows are not yet functional.
- *Extremely early prototype* – runs only on QEMU, unstable, APIs will change, no real hardware support yet. Documentation is more mature than the code.

**Documentation & code**
- Design docs (three‑tier model, rolling updates, portability): https://github.com/DslsDZC/HIC/tree/main/docs
- Repo: https://github.com/DslsDZC/HIC(the code is a work in progress – expect rough edges)

I'm particularly interested in feedback on:
- The physical contiguous allocation approach – how do you handle fragmentation? (We use a combination of static build‑time allocation, a buddy allocator with slab caches, and an emergency reserve – details in the portability doc.)
- The capability system design – any pitfalls I should be aware of?
- Suggestions for ARM/RISC‑V ports.

Thanks for reading – happy to answer any questions!

[nexos]

Looks interesting! Some similar ideas as I have for an OS

However, I do have a few comments:

- Capabilities: I think you're over-complicating them a little bit. A capability should ideally be referenced by a process-local handle, that is just a 32-bit integer or something similar. In my design, I have two main distinctions: ports and capabilities. A port is a point for communication. A capability is the ability to access that port. When a program wants access to a port, a trusted system process must grant it that capability. From their, when a program wants to use the capability, it just has a flat, 32-bit integer ID. It can then simply have a lookup table to lookup the process which it is attempting to message and it will take just a couple CPU instructions on the fast path. By complicating capability lookup at all, you're wasting CPU cycles. Let verification be done only when the capability is first accessed; from their, the process can access it without question.

- Memory allocation: I personally am not a fan of physical memory managers who attempt to allocate contiguous blocks. The reason why is that it adds a lot of complexity to what should be simple. Simply let a page allocator take a single free page from a free list; if you need to map multiple pages, take multiple pages. From a cache standpoint, implement a cache coloring scheme so you're helping distribute those pages over the cache. That's how Windows and XNU do it. Linux has a buddy allocator, but again I still feel that's slightly to complex.

Also, as a final note, one thing I've learned is OS development is that while design docs and complex design ideas are great, sometimes is better just to jump in and figure all that out as you go. Often times when you've spent hours writing in depth design docs and then go to implement, you'll find that it's nothing like you expected. As they say, make it work, make it right, and then make it fast.

Anyway keep up the good work!

[DslsDZC]

Looks interesting! Some similar ideas as I have for an OS

However, I do have a few comments:

- Capabilities: I think you're over-complicating them a little bit. A capability should ideally be referenced by a process-local handle, that is just a 32-bit integer or something similar. In my design, I have two main distinctions: ports and capabilities. A port is a point for communication. A capability is the ability to access that port. When a program wants access to a port, a trusted system process must grant it that capability. From their, when a program wants to use the capability, it just has a flat, 32-bit integer ID. It can then simply have a lookup table to lookup the process which it is attempting to message and it will take just a couple CPU instructions on the fast path. By complicating capability lookup at all, you're wasting CPU cycles. Let verification be done only when the capability is first accessed; from their, the process can access it without question.

- Memory allocation: I personally am not a fan of physical memory managers who attempt to allocate contiguous blocks. The reason why is that it adds a lot of complexity to what should be simple. Simply let a page allocator take a single free page from a free list; if you need to map multiple pages, take multiple pages. From a cache standpoint, implement a cache coloring scheme so you're helping distribute those pages over the cache. That's how Windows and XNU do it. Linux has a buddy allocator, but again I still feel that's slightly to complex.

Also, as a final note, one thing I've learned is OS development is that while design docs and complex design ideas are great, sometimes is better just to jump in and figure all that out as you go. Often times when you've spent hours writing in depth design docs and then go to implement, you'll find that it's nothing like you expected. As they say, make it work, make it right, and then make it fast.

Anyway keep up the good work!

Thank you for your thoughtful feedback—it’s always valuable to hear from someone with hands‑on OS development experience. I’d like to offer a perspective on why HIC makes the choices it does, and how they align with its core goals.

1. Capability system: why verify every time?
You’re right that a one‑time verification + trusted handle would be simpler and even faster in a static environment. However, HIC is designed for dynamic, long‑running systems where capabilities can be revoked, derived, and transferred at any time. Once a capability is revoked, any further access must be denied immediately—something a one‑time verification cannot guarantee without complex revocation notifications or versioned handles. By checking on every access (in <5ns, about 13 instructions), revocation becomes instant and trivial: the kernel simply removes the entry from the capability table, and the next access fails naturally. This tiny, predictable overhead is a deliberate trade‑off for strong, dynamic security.

2. Memory allocation: why continuous physical memory?
You’re absolutely right that a simple page allocator with cache coloring (as in Windows/XNU) is elegant and efficient for general‑purpose systems. But HIC’s foundation is physical isolation: each privileged service runs in its own contiguous physical memory region, mapped directly 1:1. This is what enables:

· Hardware‑enforced isolation (no two services share physical pages).
· Direct, same‑privilege calls between services and Core‑0 (no page‑table switches).
· Deterministic performance (no TLB misses or page faults for service memory).

If we allocated memory page‑by‑page, a service’s memory would be scattered across physical RAM, forcing us to use complex page tables and breaking the 1:1 mapping—which would reintroduce the very overheads HIC aims to eliminate. Yes, continuous allocation is more complex, but it’s the enabler of HIC’s core value proposition. We mitigate fragmentation through build‑time static planning and a dynamic pool with defragmentation strategies—so the complexity is contained, not avoided, for the sake of architectural integrity.

3. Documentation vs. implementation
I completely agree that “make it work, make it right, make it fast” is a sound mantra, and that over‑documentation can sometimes delay progress. However, HIC’s design documents aren’t just speculative—they’re the result of carefully balancing multiple conflicting requirements (security, performance, flexibility, determinism). They serve as a blueprint for formal verification (Core‑0 is kept under 10k lines so it can be mathematically proven correct) and for communicating the architecture to potential contributors. We’re not stuck in design; the code is already being written and tested. The documents guide the implementation, not replace it.

4. Current status and a small note
We believe the current design documentation is sufficiently complete to begin implementation, and we have already started coding. As we encounter real-world challenges during development, we will refine the documentation accordingly—making it a living document that evolves with the project. Also, a small note: the author is only 14 years old, so please be gentle with criticism . We welcome constructive feedback and are eager to learn and improve.

Again, thank you for taking the time to share your thoughts. It’s clear you have deep practical insight, and I’d be happy to discuss any of these points further—either here or in a more detailed thread. Your project NexNix sounds interesting too; I’ll definitely take a look!

[Octocontrabass]

But HIC’s foundation is physical isolation: each privileged service runs in its own contiguous physical memory region, mapped directly 1:1. This is what enables:

· Hardware‑enforced isolation (no two services share physical pages).

Hardware-enforced isolation works just as well without contiguous physical memory.

· Direct, same‑privilege calls between services and Core‑0 (no page‑table switches).

How much security do you want? Some CPU speculative execution security vulnerabilities can only be mitigated by switching page tables.

· Deterministic performance (no TLB misses or page faults for service memory).

You can't avoid TLB misses on x86-64.

If we allocated memory page‑by‑page, a service’s memory would be scattered across physical RAM, forcing us to use complex page tables and breaking the 1:1 mapping—which would reintroduce the very overheads HIC aims to eliminate.

No matter how you allocate memory, x86-64 forces you to use complex page tables. Larger pages can make the page tables simpler, but it will make allocation more complicated.

It sounds like some of your design choices may not be based on reality...

[DslsDZC]

But HIC’s foundation is physical isolation: each privileged service runs in its own contiguous physical memory region, mapped directly 1:1. This is what enables:

· Hardware‑enforced isolation (no two services share physical pages).

Hardware-enforced isolation works just as well without contiguous physical memory.

· Direct, same‑privilege calls between services and Core‑0 (no page‑table switches).

How much security do you want? Some CPU speculative execution security vulnerabilities can only be mitigated by switching page tables.

· Deterministic performance (no TLB misses or page faults for service memory).

You can't avoid TLB misses on x86-64.

If we allocated memory page‑by‑page, a service’s memory would be scattered across physical RAM, forcing us to use complex page tables and breaking the 1:1 mapping—which would reintroduce the very overheads HIC aims to eliminate.

No matter how you allocate memory, x86-64 forces you to use complex page tables. Larger pages can make the page tables simpler, but it will make allocation more complicated.

It sounds like some of your design choices may not be based on reality...

Thank you for your thoughtful and detailed feedback — it's exactly this kind of critical discussion that helps refine ideas. Let me address your points one by one.

1. On hardware-enforced isolation without contiguous memory
You're right that page tables can provide isolation without requiring physical contiguity. However, HIC's choice of contiguous physical regions is driven by two goals:

· Using large pages (e.g., 2MB or 1GB) to minimize TLB misses and make page tables extremely shallow. This is a practical reality on x86-64: large pages reduce TLB pressure and improve performance determinism.
· Simplifying the memory model for formal verification: contiguous regions make it easier to reason about memory safety and isolation at the hardware level.

Yes, x86-64 still requires page tables, but with large pages they become minimal — often just 2 or 3 levels instead of 4, and they rarely need to be switched.

2. On speculative execution vulnerabilities and page-table switches
This is a very valid concern. HIC's baseline design assumes that most services are trustworthy, largely because Privileged-1 services are official modules that undergo rigorous auditing — they are code-reviewed, security-tested, and digitally signed before being admitted to the official list. Moreover, all API access is strictly controlled by the capability system: each service can only invoke interfaces it has explicit capabilities for; any unauthorized attempt is blocked and logged by Core‑0. This multi‑layer defence significantly reduces the attack surface. For cases where additional mitigation is needed, HIC can adopt techniques similar to kernel page-table isolation (KPTI), but only switched in when crossing trust boundaries — not on every service call. The goal is to keep the fast path truly fast, while still allowing stronger isolation when required.

3. On TLB misses
You're absolutely correct — you can't avoid TLB misses entirely on x86-64. But HIC's use of large pages and minimal page-table switching drastically reduces their frequency. In a typical HIC system, a service's entire memory region can be covered by a single 2MB or 1GB page, so TLB misses are rare (mostly on first access or after context switches). The claim "no TLB misses" was perhaps too absolute — I should have said "predictably low TLB miss rate" or "no TLB misses for service-internal accesses after warm-up". Thank you for catching that.

4. On page-table complexity
You're right that x86-64 forces us to use page tables, regardless of allocation strategy. But HIC's approach simplifies the page-table structure itself:

· With contiguous physical memory, we can use a single large-page entry for most of a service's address space.
· This reduces the number of page-table levels and entries, making the TLB's job easier and reducing memory overhead.

Compare that to a fully paged, scattered allocation, which would require many small-page entries and deeper page tables — that's the "complexity" HIC avoids.

5. On "not based on reality"
I appreciate your skepticism — it's healthy. HIC's design choices are indeed grounded in real hardware characteristics (large pages, TLB behavior, MMU capabilities) and real-world requirements (deterministic performance, verifiable security). They may not align with every use case, but they are optimized for high-security, high-assurance environments where predictability and minimal attack surface matter more than raw memory utilization.

6. On the trustworthiness of official modules and API access control
Finally, I'd like to emphasize HIC's foundation of trust: all Privileged-1 services (including the core Module Manager) are official modules that undergo strict auditing — code review, security testing, and digital signature verification — before being included in the official list. Module loading and updates are centrally managed by the Module Manager, and every API access is tightly restricted by the capability system: any cross‑domain invocation requires the corresponding capability. This means that even if a service contains an undiscovered vulnerability, an attacker cannot abuse unauthorised APIs or access other services' memory. It is this combination of least privilege + mandatory auditing that makes the baseline assumption ("most services are trustworthy") practical in real deployments.

I'd be happy to discuss this further — either here, on GitHub, or wherever you prefer. Your insights are valuable, and I'd love to hear more about your own experiences with OS design.

[nexos]

On hardware-enforced isolation without contiguous memory
You're right that page tables can provide isolation without requiring physical contiguity. However, HIC's choice of contiguous physical regions is driven by two goals:

· Using large pages (e.g., 2MB or 1GB) to minimize TLB misses and make page tables extremely shallow. This is a practical reality on x86-64: large pages reduce TLB pressure and improve performance determinism.
· Simplifying the memory model for formal verification: contiguous regions make it easier to reason about memory safety and isolation at the hardware level.

That’s a very valid point. However my suggestion would be to either do:

1. A Slab allocator. I haven’t fully ironed out the details of how that would work but I know that’s how Netbsd allocates contiguous pages
2. A cache of large pages. For example, keep a particular number of pages in the system large and small and allocate from a pool depending on needs. Simple, but not dynamic.

Using a bitmap like (I believe but could be wrong) your using is OK, but tends to be quite inefficient and has very poor worst case performance. A slab has worse performance then a simple pool, but it’s still very well bounded

[Octocontrabass]

· Using large pages (e.g., 2MB or 1GB) to minimize TLB misses and make page tables extremely shallow. This is a practical reality on x86-64: large pages reduce TLB pressure and improve performance determinism.

Large pages come with their own complexity. They must be aligned. They must not span multiple memory types, so you'll need to check the MTRRs. You still need 4kiB allocations for the page tables, since those are always 4kiB. Not all x86-64 CPUs support 1GiB pages, so you must check before you can use them. These are not insurmountable problems, but they are all problems you must plan for.

· Simplifying the memory model for formal verification: contiguous regions make it easier to reason about memory safety and isolation at the hardware level.

I don't see how it would make any difference.

Yes, x86-64 still requires page tables, but with large pages they become minimal — often just 2 or 3 levels instead of 4,

There's no "often" about it, there are 2 levels when you use 1GiB pages and there are 3 levels when you use 2MiB pages.

and they rarely need to be switched.

Page size has nothing to do with that. When you need to switch address spaces, you need to switch page tables.

HIC's use of large pages

But HIC doesn't use large pages.