x86-64 OS development from Windows with ChatGPT assistance — real hardware and spec-driven.

[Norppa]

Introduction

I've been doing OS development in NASM on and off for years.

Recently, I restarted an x86-64 hobby OS project from scratch, with one deliberate difference: I'm using an LLM as a development partner.

This is not an "AI generated an OS in five minutes" project.

The project uses architecture and platform specifications to define expected behaviour, and physical hardware to validate the implementation.

My basic rules are:

• Check hardware-facing structures, offsets and bitfields against specifications or reliable technical references.
• Validate each development step before moving on.
• Generated code is not accepted simply because it compiles.
• Investigate and correct failures before continuing.
• Avoid giant generated code dumps containing several unfinished subsystems.

Development environment and hardware

The entire build process currently runs on Windows 11, using NASM and LLVM's lld-link to produce PE32+ x86-64 UEFI executables.

No Linux host is involved.

The physical development machine is:

Code: Select all

CPU:         AMD Ryzen 5 9600X
             Zen 5 / Granite Ridge

Motherboard: ASUS TUF GAMING B850-PLUS WIFI

RAM:         32 GiB

Graphics:    AMD integrated graphics
             NVIDIA GeForce RTX 5050

Boot:        Native UEFI

Language choices

The current system is still written primarily in NASM.

I originally wanted to write the entire OS in assembly. Over time, however, I found myself building increasingly elaborate NASM macros to express structures that were no longer especially assembly-specific.

That changed how I think about language choice.

Assembly is particularly useful when the exact machine operations are part of the problem being understood. For other subsystems, the structure of the problem may eventually matter more than the exact instruction sequence.

So my current rule is:

Use assembly when the exact machine operations are part of the problem being understood.

Use a higher-level language only when assembly can no longer express the problem clearly without adding unnecessary machinery.

For more structurally complex subsystems, the language will be chosen module by module rather than imposing an "assembly at all costs" rule.


Milestone 1 — UEFI GOP framebuffer

Status: PASS on physical hardware

Validated path:

Code: Select all

ASUS UEFI
    |
    v
\EFI\BOOT\BOOTX64.EFI (PE32+ x86-64)
    |
    v
NASM entry point
    |
    v
EFI_BOOT_SERVICES
    |
    v
LocateProtocol()
    |
    v
Graphics Output Protocol (GOP)
    |
    v
Framebuffer information
    |
    v
Direct pixel writes

Expected display

The firmware display is replaced by a full-screen dark gray background with a small fixed white triangular software cursor near the upper-left corner:

Code: Select all

┌──────────────────────────────┐
│ dark gray background         │
│                              │
│     ◢ small white cursor     │
│                              │
│                              │
└──────────────────────────────┘

The cursor is drawn pixel by pixel; the character above only approximates its shape and orientation.

It is intentionally primitive and serves as a rendering test. There is no keyboard or mouse input yet.


Booting on physical hardware

There is no installer.

The OS boots as a standard UEFI removable-media application from a dedicated USB flash drive and does not access the internal SSD containing Windows.

Until the boot path is stable, I prefer keeping it separate from the Windows EFI System Partition.

Breaking a disposable USB flash drive is considerably funnier than breaking Windows Boot Manager.


1. Prepare the USB drive

Format a spare USB flash drive as FAT32, create the standard x86-64 UEFI removable-media directory, and copy the generated executable as:

Code: Select all

\EFI\BOOT\BOOTX64.EFI

For example, assuming the USB drive is X: on Windows:

Code: Select all

mkdir X:\EFI\BOOT
copy /Y build\BOOTX64.EFI X:\EFI\BOOT\BOOTX64.EFI

No MBR boot sector or Legacy BIOS boot path is involved.


2. Configure the firmware

My development configuration is:

Code: Select all

UEFI         ON
CSM          OFF
Secure Boot  OFF
TPM          ON, but unused by the OS

Secure Boot is disabled because the development EFI executable is unsigned.

TPM remains enabled but is currently ignored by the OS.


ASUS configuration example

On my ASUS TUF GAMING B850-PLUS WIFI, the relevant setting is:

Code: Select all

UEFI Setup
 -> Boot
 -> Secure Boot
 -> OS Type

ASUS documents the choices as:

Code: Select all

Windows UEFI mode -> Secure Boot enabled
Other OS          -> Secure Boot disabled

For development I selected Other OS.

Windows System Information (msinfo32) reports:

Code: Select all

BIOS Mode:         UEFI
Secure Boot State: Off

Disabling Secure Boot does not imply switching to Legacy BIOS or enabling CSM, and there is no need to clear the Secure Boot key database merely to test this unsigned hobby OS.

ASUS — How to Enable/Disable Secure Boot


Source code

I've attached the complete source code for this stage, including:

• NASM source and UEFI definitions
• Windows build script and PE32+ EFI generation
• GOP framebuffer access
• Fixed software cursor rendering

OperatingSystem.zip

(6.88 KiB) Downloaded 19 times

Next milestone — Leaving Boot Services

The next target is:

Code: Select all

GetMemoryMap()
    |
    v
Prepare boot information
    |
    v
ExitBootServices()
    |
    v
Continue executing independently
    |
    v
Keep rendering to the existing framebuffer

This is where the project begins transitioning from an EFI application into an autonomous operating-system environment.


Roadmap

The current direction is approximately:

Code: Select all

UEFI
 |
 +-- GOP / framebuffer                    [PASS]
 |
 +-- GetMemoryMap / ExitBootServices      [NEXT]
 |
 +-- CPU / memory / interrupts
 |
 +-- ACPI
 |    |
 |    +-- RSDP
 |    +-- XSDT
 |    +-- MCFG
 |
 +-- PCI Express
 |    |
 |    +-- ECAM
 |    +-- bus/device/function enumeration
 |    +-- Vendor ID / Device ID
 |    +-- Class / Subclass / ProgIF
 |    +-- BAR / MMIO discovery
 |
 +-- Review PCIe / BAR foundation
 |
 +-- xHCI
 |    |
 |    +-- USB
 |    +-- HID
 |    +-- keyboard
 |    +-- mouse
 |
 +-- Movable software cursor
 |
 +-- Eventually, a graphical environment

[Norppa]

OperatingSystem.zip

(15.83 KiB) Downloaded 12 times

Milestone 3: custom GDT, IDT and first exception handler working.
Small update, but this one is more interesting than the framebuffer test.

The kernel now leaves the UEFI execution environment and establishes its own minimal CPU environment.

Current boot path:

Code: Select all

UEFI
  ↓
GOP framebuffer
  ↓
GetMemoryMap()
  ↓
ExitBootServices()
  ↓
private kernel stack
  ↓
custom GDT
  ↓
reload CS / SS
  ↓
custom IDT
  ↓
INT3
  ↓
custom #BP handler
  ↓
IRETQ
  ↓
kernel resumes execution

For the test I deliberately use four framebuffer markers:

Code: Select all

green    = ExitBootServices succeeded and private stack is active
magenta  = custom GDT loaded and CS/SS successfully reloaded
white    = #BP handler was entered through my own IDT
gray     = IRETQ returned successfully to the instruction after INT3

All four markers appear on the machine.

The IDT currently contains 256 16-byte entries. All vectors initially point to a generic fatal handler, with vector 3 replaced by a dedicated breakpoint handler.

The breakpoint test is simply:

Code: Select all

    lidt    [IdtDescriptor]

    int3

    ; Reached only if the #BP handler successfully returns with IRETQ.

The handler saves the general-purpose registers, draws the white marker, restores the registers and returns:

Code: Select all

isr_breakpoint:
    PUSH_GPRS

    ; framebuffer diagnostic marker
    ...

    POP_GPRS
    iretq

Interrupts are still disabled (IF=0). I am intentionally not enabling hardware IRQs yet; there is no APIC/interrupt-controller initialization at this stage.

The GDT currently contains only:

Code: Select all

0x00  null descriptor
0x08  64-bit ring-0 code
0x10  ring-0 data

After loading it with LGDT, CS is reloaded with a far return and the data/stack segment selectors are replaced as well.

I also switched away from the stack supplied by the firmware and use a 64 KiB stack contained in the kernel image.

The next step will probably be proper CPU exception stubs for vectors 0–31, including normalization of exceptions with/without error codes, so that a panic handler can display at least:

Code: Select all

vector
error code
RIP

After that I intend to start using the UEFI memory map for physical-memory management before moving on to ACPI/PCIe.

[Norppa]

OperatingSystem.zip

(20.03 KiB) Downloaded 13 times

Milestone 5 completed.
This milestone introduces the project's first physical memory manager (mm\physical.asm).

The kernel now keeps the UEFI memory map, parses it after ExitBootServices(), and builds a simple 4 KiB page allocator from EfiConventionalMemory. For the moment, memory below 1 MiB is intentionally excluded, and freed pages are recycled through a small LIFO free list.

The current self-test checks:
- descriptor parsing
- usable memory accounting
- allocation of multiple distinct pages
- direct write/read access to allocated pages
- free/reallocate correctness
- restoration of the original free-page count

On the current test system, the allocator reports:
- 0x3E memory descriptors
- 0x7C158E000 managed bytes (32 GB RAM)
- 0x7C158E free pages initially

The test then allocates:
- A = 0x00100000
- B = 0x00101000
- C = 0x00102000

After freeing B, the next allocation returns:
- D = 0x00101000

So the recycled-page path behaves as expected. After returning all test pages, the final free-page count again matches the initial one.

So the project now has working:
- UEFI bootstrapping
- framebuffer output
- ExitBootServices()
- private stack setup
- GDT / IDT
- CPU exception handling
- basic physical page allocation

Next milestone: build paging structures and switch to CR3.

[Norppa]

OperatingSystem.zip

(32.45 KiB) Downloaded 11 times

Milestone 6 completed, and Milestone 7 is now partially implemented.

This update replaces the firmware paging environment with page tables owned by HobbyOS and introduces the first virtual-memory primitives.


Milestone 6 — HobbyOS-owned paging
-----------------------------------

The kernel now:

- inspects the paging state inherited from UEFI
- allocates and clears its own paging structures
- builds a bootstrap identity map
- maps the GOP framebuffer when required
- verifies all critical kernel ranges before switching
- loads its own PML4 through CR3
- continues executing and drawing to the framebuffer afterwards
- verifies that the PMM still works under the new CR3

The current bootstrap map covers:

[0, 6 GiB)

using 2 MiB pages.

The PMM is temporarily restricted to conventional RAM inside:

[1 MiB, 6 GiB)

so that all pages returned by the allocator remain directly accessible during
the current bootstrap stage.

This restriction is temporary and will be revisited by the virtual-memory
manager.

On the current Ryzen 5 9600X test system, the CR3 transition was:

firmware CR3 : 0x78001000
HobbyOS CR3 : 0x00101000

The post-switch PMM regression also verifies both fresh sequential allocation
and recycled-page allocation while HobbyOS page tables are active.


Milestone 7 — Virtual memory
----------------------------

The project now explicitly distinguishes physical and virtual addresses.

The following primitives are working:

Code: Select all

    paging_translate()
    paging_map_page()
    paging_unmap_page()
    paging_map_range()

7A — Translation
The kernel can walk its own page tables and translate mapped virtual addresses
back to physical addresses, while rejecting unmapped and non-canonical ones.

7B — 4 KiB mapping
The kernel can create explicit non-identity 4 KiB mappings and verify that
writes through the virtual address reach the expected physical page.

7C — Unmapping
The kernel can remove a 4 KiB mapping, invalidate the TLB entry with INVLPG,
and return the original physical address without freeing the physical page.

7D — Range mapping
The kernel can map and later remove contiguous virtual ranges.

The current hardware test maps three pages:

physical start : 0x0000000140008000
virtual start : 0x0000010000004000

The test verifies translation, read/write aliasing, cleanup, and reports:

VmmRangeChecks = 0x1FF

All nine checks passed on real hardware.

The current memory path is therefore:

Code: Select all

    PMM
        -> physical page ownership

    paging/VMM
        -> virtual mappings

and physical addresses are no longer assumed to be directly interchangeable
with virtual addresses as a general rule.

The next architectural question is how the kernel should access ordinary
physical RAM permanently.

The main candidates are a physical direct map or temporary mappings or a hybrid strategy

Current working milestones:

- UEFI bootstrapping
- GOP framebuffer
- ExitBootServices()
- private kernel stack
- GDT / IDT
- CPU exception handling
- physical memory manager
- HobbyOS-owned paging / CR3
- page-table translation
- 4 KiB map/unmap
- range mapping

[Norppa]

OperatingSystem.zip

(61.32 KiB) Downloaded 9 times

Milestone 7 completed — virtual memory foundation

Since the previous update, HobbyOS gained:

Code: Select all

4 KiB map / query / unmap
2 MiB map / query / unmap
range mapping
virtual -> physical translation
permanent high-half physical direct map
semantic mapping properties
WB / UC cache policy
PMM access above the old 6 GiB bootstrap ceiling

The direct map lives at:

Code: Select all

0xFFFF800000000000

and uses 2 MiB leaves where possible, with 4 KiB fallback.
Mapping policy is now expressed with HobbyOS-level properties:

Code: Select all

MAP_WRITABLE
MAP_EXECUTABLE
MAP_USER

CACHE_WRITE_BACK
CACHE_UNCACHEABLE

rather than exposing raw x86 paging bits to callers.
The kernel also explicitly establishes the paging-related CPU state it depends on, including write protection and NX support.
All of this has been validated on real hardware for both 4 KiB and 2 MiB mappings.
Current state:

Code: Select all

M7A  translation                    PASS
M7B  4 KiB mapping                  PASS
M7C  4 KiB unmapping                PASS
M7D  range mapping                  PASS
M7E  permanent physical direct map  PASS
M7F  remove 6 GiB PMM ceiling       PASS
M7G  mapping policy                 PASS

M7   virtual memory foundation      PASS

1 GiB leaves are detected but intentionally deferred for now.

Bonus: Paging Playground
I also added a small development-only paging_playground.asm.
The first demo maps two virtual addresses to the same physical page, verifies aliasing, then remaps one of them to a different physical page.

Code: Select all

VA_A ----+
         +----> Physical A
VA_B ----+

then:

Code: Select all

VA_A --------> Physical A
VA_B --------> Physical B

All 13 checks pass on hardware:

Code: Select all

PagingPlaygroundChecks = 0x1FFF

It is not part of the normal boot path; it is just a place to experiment with paging mechanisms before they become higher-level architecture.

Next:

Code: Select all

M8 — ACPI

[dseller]

What are you even talking about man? What do all these words mean that you have generated through your LLM?

[Norppa]

What are you even talking about man? What do all these words mean that you have generated through your LLM?

Yes. I use the LLM through long discussions, back-and-forth, and technical debate over hours.
Neither side is just prescribing an end-to-end solution; it’s an iterative exchange.

[dseller]

What are you even talking about man? What do all these words mean that you have generated through your LLM?

Yes. I use the LLM through long discussions, back-and-forth, and technical debate over hours.
Neither side is just prescribing an end-to-end solution; it’s an iterative exchange.

Explain to me, without consulting your LLM, what your “semantic mapping properties” feature does

[Norppa]

Explain to me, without consulting your LLM, what your “semantic mapping properties” feature does

I’m not going to answer under a “without consulting your LLM” condition — using it is part of how I work, even when I’m doing technical reasoning with other humans.

It came out of a discussion about how Windows and Linux handle memory protection without exposing raw x86 page-table bits everywhere.

HobbyOS callers express intent; the paging backend handles the hardware encoding.

I think of understanding a bit like foveated rendering: I keep high resolution on the part of the system I’m actively working on, and a lower-resolution architectural model of the rest until I need to zoom back in.

[dseller]

without exposing raw x86 page-table bits everywhere.

What do you mean?

[Norppa]

What do you mean?

I'm not going to trace every branch down to the raw x86 page-table bits here — LLMs exist, and anyone who wants that level of detail can use them to drill further.

By "not exposing raw x86 page-table bits everywhere", what I mean is that I’m dealing with the same page-table concepts these systems use:

WINDOWS

VirtualProtect() takes semantic protection values such as PAGE_READWRITE / PAGE_EXECUTE_READ / ...

Microsoft documents that Windows uses virtual-memory hardware for memory protection, while page tables perform the virtual-to-physical translation.


LINUX

mprotect() accepts PROT_READ / PROT_WRITE / PROT_EXEC.

Those are translated by calc_vm_prot_bits() into Linux VM_READ / VM_WRITE / VM_EXEC state.

mprotect_fixup() updates the VMA protection and drives the protection change.

The x86 backend then has its own protection_map[] mapping Linux VM semantics to architecture-specific page protections.

For existing mappings, the path reaches change_pte_range() / pte_modify().

Underneath that are the actual x86 bits _PAGE_RW, _PAGE_USER, _PAGE_PWT, _PAGE_PCD, _PAGE_NX, ...


So, conceptually:

Windows:
VirtualProtect(PAGE_*)
→ Windows memory-protection model
→ PTE/PDE
→ x86 hardware

Linux:
mprotect(PROT_*)
→ VM_* protection model
→ pgprot / PTE update
→ _PAGE_* x86 encoding
→ x86 hardware

HobbyOS:
MAP_WRITABLE / MAP_EXECUTABLE / MAP_USER / CACHE_*
→ x86-64 paging backend
→ PAGE_WRITE / PAGE_NX / PAGE_USER / PAT...
→ x86 hardware


That's what I meant by "not exposing raw x86 page-table bits everywhere":

the higher layer describes what the mapping is supposed to mean;
the architecture backend owns how that meaning is encoded for the hardware.



P.S. Microsoft's driver documentation gets even closer to the metal — it explicitly talks about CPU page tables, PTEs, and virtual-to-physical translation while explaining DMA map registers:

• Map Registers — describes the CPU translating virtual addresses to physical addresses through page tables, with each PTE mapping a virtual page to a physical page, then compares that translation model with DMA map registers.
  
  !pte — displays the actual PTE/PDE for a virtual address and decodes PFNs and status bits.
  
  !vtop — converts a virtual address to the corresponding physical address and shows page-table/PFN information.
  
  MmGetPhysicalAddress() — returns the physical address corresponding to a valid nonpaged virtual address.
  
  MmMapIoSpace() — maps a physical address range into system virtual address space with a specified cache policy.

[iansjack]

I’m not going to answer under a “without consulting your LLM” condition — using it is part of how I work, even when I’m doing technical reasoning with other humans.

That certainly explains why your posts are so incomprehensible.

[Norppa]

That certainly explains why your posts are so incomprehensible.

If there’s a specific technical point you find incomprehensible, quote it and I’ll try to explain it.
“Your posts are incomprehensible” doesn’t give me much to work with.

[iansjack]

There's little point in that as you would only consult your LLM for an answer. So it would just be a circular process.

Truth is I have little interest in AI generated software. I do this stuff for fun.

[nullplan]

Truth is I have little interest in AI generated software. I do this stuff for fun.

I also don't quite understand automating away the most engaging part of this hobby. Actually writing the code and seeing the program gain features and lose bugs is what I'm here for.