OSDev.org

The Place to Start for Operating System Developers
It is currently Tue Jun 27, 2017 3:05 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: OS/Z and BOOTBOOT Protocol
PostPosted: Thu Oct 13, 2016 6:51 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 85
Hi there!

After a couple years gap, I've started os dev again. I've started from scratch, and after a few weeks my new OS reached the first milestone, it can boot! :-) It can't do much more than drawing color boxes on screen, but this post isn't about OS/Z (not to confuse with z/OS). It's about the boot loader.

Have you ever wondered how good it would be if the firmware'd set up long mode, load your initrd and start executing your kernel right away? It was just a dream, you had to roll your own loader or learn to use one. But BIOS is ancient, needs lots of assembly; GRUB is a beast, almost an OS on it's own; and EFI is a real interface nightmare. Who want to struggle with all that stuff just to start the kernel?

I've solved that in an elegant way: my boot loader resides in ROM. On boot, it will locate the kernel text segment in the first ELF64 executable on the first init ramdisk on the first bootable partition of the first drive (hope you can follow :-)). It's doing it by looking for file magic, so file system independently. After that it sets up a minimal, well defined long mode environment (with framebuffer) and calls the kernel's entry point. The whole procedure is described in detail at

BOOTBOOT Protocol

(the name refers to 64 bits) The reference implementations for BIOS and EFI can be found there as well. My basic principle is K.I.S.S., so the code base is minimal with the least dependencies possible.

And the best part is, it's Public Domain! So you can use those ROM images to boot your own kernel from your own filesystem without the need of writing an fs driver! Never been easier to start OS development! They work with qemu, bochs and real hardware as well (at your own risk). If you don't want to mess with ROMs, you can boot the loader from disk too, as it supports:

- Standard MBR booting on GPT disks (real mode)
- MultiBoot (prot mode)
- BIOS Expansion ROM (real mode)
- UEFI OS loader application (long mode)
- PCI Option ROM (long mode)

It doesn't matter which one is used, your kernel will start with the same, well-known long mode environment.

Tested with FAT12, cpio, tar and FS/Z (not to confuse with Zfs) initrd formats on FAT12, FAT16 and FAT32 EFI System Partitions.

Hope it helps somebody,
bzt

ps.: any testing result would be appreciated! Let me know how it worked for you!


Last edited by bzt on Thu Dec 29, 2016 3:44 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Dec 28, 2016 9:24 pm 
Offline
Member
Member
User avatar

Joined: Sun Dec 25, 2016 1:54 am
Posts: 195
sorry... why is it great with gpl v3?

_________________
Plagiarize. Plagiarize. Let not one line escape thine eyes...


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Thu Dec 29, 2016 3:18 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 85
dchapiesky wrote:
sorry... why is it great with gpl v3?

I wanted it to be permissive. Just to make sure I've re-licensed BOOTBOOT as Public Domain.
The permission to use my loader in any way you wish is hereby granted.

The licensing terms of OS/Z hasn't changed.

Bests,
bzt


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Sun Jun 04, 2017 3:55 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 85
I proudly announce the first stable version of BOOTBOOT loader! It's finally done and bugfixed! :-)

Files
Download and source
* mbr.bin - 512 byte stage 1 loader, works as MBR and as a chainloaded VBR
* bootboot.bin - 8192 byte stage 2 loader, can be booted with mbr.bin, a Multiboot compliant loader, or as a BIOS Expansion ROM
* bootboot.efi - UEFI version of the loader, with -? argument gives a little usage help
* bootboot.h - C header file to include in your kernel

How to use
1. create an x86_64 elf kernel (include bootboot.h in it)
2. put it along with other files in an archive. The format doesn't matter at all, just make sure your kernel is the first elf executable inside. Use your own filesystem if you like! Optionally you can compress it with gzip.
3. create a GPT disk image with a small, bootable FAT12/16/32 partition on it (attribute bit 2 set or use ESP's GUID).
4. create a BOOTBOOT directory on that partition, and copy your archive as INITRD there.

Now you can directly boot up your OS from that disk with bootboot.bin in ROM (qemu: -option-rom argument, bochs: optionromimageX config variable), no blessing or setup required at all! :-)

Passing parameters
Simply create a text file holding "key=value" pairs at FS0:\BOOTBOOT\CONFIG. Fill up with newline characters to be exactly 4096 bytes (1 page) long. The loader parses only 3 keys: width and height for framebuffer size, and kernel to specify an elf executable inside initrd (only if you choose to use cpio or tar archives as initrd. For all the other formats the first elf will be loaded). All other keys are up to you and should be parsed by your kernel. Here's an example configuration file for OS/Z.

Machine state on boot
When your elf kernel gains control, the CPU is already in long mode, with IRQs disabled and interrupts turned off. It will be mapped at the highest page directory (-2M...0 if you like). All information gathered in a platform independent structure, like boot time, memory map, framebuffer characteristics, ACPI and SMBI pointers etc. (see bootboot.h). Detailed description and an example kernel with linker script is also provided on github.

BOOTBOOT hides all the tricky and boring details of the early boot process (A20, E820, VBE, LocateProtocol, GOP->QueryMode etc.), so you can focus on writing your kernel.

Tested with
Standard (mbr.bin): qemu 2.9.0, bochs 2.6.9, virtualbox 5.1.22 and real hardware (on USB stick and HDD)
Option ROM: qemu, bochs
UEFI: qemu (OVMF r15214), virtualbox


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Jun 07, 2017 9:45 am 
Offline
Member
Member
User avatar

Joined: Fri Oct 27, 2006 9:42 am
Posts: 871
Location: Athens, GA, USA
OK, I am... puzzled as to why this is a thing. I mean, in a virtual machine, yes, it sounds great, but it rather leaves you stuck once you move to live hardware. I suspect I am missing something, and would appreciate it if you clarified things a bit.

The whole point of an option ROM is that it comes as part of the PCI device it is with; which device's option ROM would you use? The only one that would make sense would be that of the drive you are booting from itself - but most modern SATA drive connectors bypass the PCI bus, and most drives don't have any system-accessible ROM anyway (the on-drive controllers have their own processors, meaning that each drive is, in effect, a co-processor) even if the drive has any flashable ROM ( better-quality ones do, but many cheaper ones just use masked ROM and leave you hanging if there is a bug), you still couldn't use it.

The pending rise of non-volatile memory systems such as 3D XPoint and MRAM will probably quash even that, since eventually the goal is not so much to use it for SSDs, but as part of the main memory. It is very likely that this will put an end to the concept of a separate boot BIOS (in the general sense that covers both PC-BIOS and UEFI) entirely, as things move more towards a persistence-based model in which the OS and session data are resident in non-volatile memory, with the kernel hot-swapped for most updates, and only rebooted in the case of a catastrophic failure - sort of like a return to the days of core memory.

(There's a famous story about the PDP-1 and PDP-6: the factory techs would use Spacewar! as the final shakedown for each new computer, as it was the one non-test program they had that exercised every part of the system. Then, because core was non-volatile, it would be ready to run the next time it was booted, so the installation techs would also then use the game to test the system in the field. =D> )

Anyway, my point is that when and if that becomes commonplace, I would expect that in day-to-day operation the startup sequence would be dominated by something like a CRC performed on the kernel rather than on loading the OS. The system loader itself would probably be just a protected section of NVM which could only be updated with a special permission, and would simply hold the startup code for a kernel (or, more likely, a rump hypervisor that in turn would be able to start one or more kernels and manage switching between them).

So, as I said, I think I must be missing something with all this.

_________________
Rev. First Speaker Schol-R-LEA;2 LCF ELF JAM POEE KoR KCO PPWMTF
μή εἶναι βασιλικήν ἀτραπόν ἐπί γεωμετρίαν
Lisp programmers tend to seem very odd to outsiders, just like anyone else who has had a religious experience they can't quite explain to others.


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Jun 07, 2017 4:41 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 7978
Location: At his keyboard!
Hi,

Schol-R-LEA wrote:
OK, I am... puzzled as to why this is a thing. I mean, in a virtual machine, yes, it sounds great, but it rather leaves you stuck once you move to live hardware. I suspect I am missing something, and would appreciate it if you clarified things a bit.

The whole point of an option ROM is that it comes as part of the PCI device it is with; which device's option ROM would you use?


I'd assume network card ROM would be the most common.

Schol-R-LEA wrote:
The pending rise of non-volatile memory systems such as 3D XPoint and MRAM will probably quash even that, since eventually the goal is not so much to use it for SSDs, but as part of the main memory. It is very likely that this will put an end to the concept of a separate boot BIOS (in the general sense that covers both PC-BIOS and UEFI) entirely, as things move more towards a persistence-based model in which the OS and session data are resident in non-volatile memory, with the kernel hot-swapped for most updates, and only rebooted in the case of a catastrophic failure - sort of like a return to the days of core memory.


The entire concept of "persistent OS in non-volatile system memory" is completely broken and unusable; specifically because there's no way to determine how much data got trashed between the previous "shutdown" and booting it again. To guard against this you need to fence off "safe for persistent data" areas and apply some sort of standardised partitioning convention (and as soon as you do that you end up with "boring old storage device where the only difference is that it just happens to be mapped into the physical address space"). You'd also need some way to say "this data should never be persisted" (for security reasons - e.g. to ensure secure deletion of passwords, temporarily decrypted data, etc; that might remain in memory when power is turned off at exactly the wrong time).

Then there's the fact that, after 60 years of trying (and failing) to fix the "almost all software is full of bugs" problem, the first thing everyone does when things go wrong is reboot to flush out everything that got mangled and restart the system from a clean slate.

Schol-R-LEA wrote:
Anyway, my point is that when and if that becomes commonplace, I would expect that in day-to-day operation the startup sequence would be dominated by something like a CRC performed on the kernel rather than on loading the OS. The system loader itself would probably be just a protected section of NVM which could only be updated with a special permission, and would simply hold the startup code for a kernel (or, more likely, a rump hypervisor that in turn would be able to start one or more kernels and manage switching between them).

So, as I said, I think I must be missing something with all this.


You seem to have forgotten that "non-volatile system memory" doesn't actually exist now (and if it was released tomorrow it'd take at least 10 years before it becomes ubiquitous).



Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Thu Jun 08, 2017 10:19 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 85
@Schol-R-LEA: Yes, Brendan is right, most likely it would be a network card's ROM. But not only, you can put that image on the motherboard with flashrom for example. Coreboot supports additional option ROM payloads (should work, but not tested so far).
The point is, it's easy to use in simulators; you can create a hardware to support it directly; but if you don't want to, you can still use it from disk (either with BIOS or with UEFI). You have a lot of options to boot the same long mode environment, and no configuration needed. It's useful when you rapidly changing your kernel or initrd (which is a common case during development). Honestly I don't like GRUB2, it's way to over-complicated to my taste, and it's not able to boot a kernel from my filesystem. Also it loads kernel and initrd as separate files (which is common for a monolithic design), whilst BOOTBOOT loads the kernel from the initrd (more suitable for a microkernel architecture imho).

And about presistent OS, I would really like to see it, but still, it must have an option to do a clear reboot (just to reset the whole system).


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group