Binary execution doesn't work

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
Post Reply
bomboclaat954
Posts: 3
Joined: Sat Jun 06, 2026 5:29 am
Libera.chat IRC: bomboclaat954

Binary execution doesn't work

Post by bomboclaat954 »

I wrote simple syscalls and one of them is responsible for executing binary code (raw bytes, opcodes). I also have very simplified initramfs. And on the init side I do something like this (code below) just to check if the string will appear.

Code: Select all

#include <bomboclaat.h>
#include <stdint.h>
#include <vfs.h>

extern int shell_main();

void syscall_exec(vfs_file_t *exe)
{
    __asm__ volatile(
        "int $0x80"
        :
        : "a"(2), "D"(exe)
        : "memory");
}

int _init_start()
{
    vfs_file_t exe;
    uint8_t exe_content[] = {
        0xB8, 0x01, 0x00, 0x00, 0x00,
        0x48, 0xBF, 0x13, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00,
        0xCD, 0x80,
        0xEB, 0xFE,
        'B', 'O', 'M', 'B', 'O', 'C', 'L', 'A', 'A', 'T', ' ', 'K', 'e', 'r', 'n', 'e', 'l', '\n', 0};

    exe.content = exe_content;
    exe.size = sizeof(exe_content);
    syscall_exec(&exe);
    printf("");
    shell_main();
    while (1)
        ;
}
On the kernel side i have this:

Code: Select all

        //HHDM offset = 0, so I skipped it here (it caused page fault)
        vmm_table_t *new_vmm = vmm_init();
        vfs_file_t *executable = (vfs_file_t *)r->rdi;

        size_t bytes_left = executable->size;
        uint8_t *src = (uint8_t *)executable->content;

        for (size_t offset = 0; offset < executable->size; offset += PAGE_SIZE)
        {
            void *phys_frame = pmm_alloc_frame();
            if (!phys_frame)
                panic("PMM out of frames", 0, 0);
            size_t chunk_size = (bytes_left > PAGE_SIZE) ? PAGE_SIZE : bytes_left;
            uint8_t *dst = (uint8_t *)phys_frame;
            memcpy(dst, src, chunk_size);
            vmm_map_page(new_vmm, PROGRAM_START + offset, (uintptr_t)phys_frame, VMM_PRESENT | VMM_WRITE);
            src += chunk_size;
            bytes_left -= chunk_size;
        }

        uintptr_t new_pml4_phys = (uintptr_t)new_vmm;

        asm volatile("mov %0, %%cr3" ::"r"(new_pml4_phys) : "memory");
        void (*program)(void) = (void (*)(void))PROGRAM_START;
        program();

        return 1;
VFS file structure:

Code: Select all

struct vfs_file
{
    char *name;
    vfs_dir_t *dir;
    uint64_t size;
    uint8_t *content;
} typedef vfs_file_t;
And it should work but it doesn't. System hangs and the string doesn't show up. The good news is that there's not a triple fault because it doesn't automatically reboot.
Octocontrabass
Member
Member
Posts: 6275
Joined: Mon Mar 25, 2013 7:01 pm

Re: Binary execution doesn't work

Post by Octocontrabass »

bomboclaat954 wrote: ↑Sat Jun 06, 2026 5:39 amSystem hangs and the string doesn't show up.
What does your debugger say the CPU is doing when it hangs?
Post Reply