OSDev.org

The Place to Start for Operating System Developers
It is currently Fri Sep 22, 2017 12:13 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: OS/Z and BOOTBOOT Protocol
PostPosted: Thu Oct 13, 2016 6:51 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 138
Hi there!

After a couple years gap, I've started os dev again. I've started from scratch, and after a few weeks my new OS reached the first milestone, it can boot! :-) It can't do much more than drawing color boxes on screen, but this post isn't about OS/Z (not to confuse with z/OS). It's about the boot loader.

Have you ever wondered how good it would be if the firmware'd set up long mode, load your initrd and start executing your kernel right away? It was just a dream, you had to roll your own loader or learn to use one. But BIOS is ancient, needs lots of assembly; GRUB is a beast, almost an OS on it's own; and EFI is a real interface nightmare. Who want to struggle with all that stuff just to start the kernel?

I've solved that in an elegant way: my boot loader resides in ROM. On boot, it will locate the kernel text segment in the first ELF64 executable on the first init ramdisk on the first bootable partition of the first drive (hope you can follow :-)). It's doing it by looking for file magic, so file system independently. After that it sets up a minimal, well defined long mode environment (with framebuffer) and calls the kernel's entry point. The whole procedure is described in detail at

BOOTBOOT Protocol

(the name refers to 64 bits) The reference implementations for BIOS and EFI can be found there as well. My basic principle is K.I.S.S., so the code base is minimal with the least dependencies possible.

And the best part is, it's Public Domain! So you can use those ROM images to boot your own kernel from your own filesystem without the need of writing an fs driver! Never been easier to start OS development! They work with qemu, bochs and real hardware as well (at your own risk). If you don't want to mess with ROMs, you can boot the loader from disk too, as it supports:

- Standard MBR booting on GPT disks (real mode)
- MultiBoot (prot mode)
- BIOS Expansion ROM (real mode)
- UEFI OS loader application (long mode)
- PCI Option ROM (long mode)

It doesn't matter which one is used, your kernel will start with the same, well-known long mode environment.

Tested with FAT12, cpio, tar and FS/Z (not to confuse with Zfs) initrd formats on FAT12, FAT16 and FAT32 EFI System Partitions.

Hope it helps somebody,
bzt

ps.: any testing result would be appreciated! Let me know how it worked for you!


Last edited by bzt on Thu Dec 29, 2016 3:44 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Dec 28, 2016 9:24 pm 
Offline
Member
Member
User avatar

Joined: Sun Dec 25, 2016 1:54 am
Posts: 205
sorry... why is it great with gpl v3?

_________________
Plagiarize. Plagiarize. Let not one line escape thine eyes...


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Thu Dec 29, 2016 3:18 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 138
dchapiesky wrote:
sorry... why is it great with gpl v3?

I wanted it to be permissive. Just to make sure I've re-licensed BOOTBOOT as Public Domain.
The permission to use my loader in any way you wish is hereby granted.

The licensing terms of OS/Z hasn't changed.

Bests,
bzt


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Sun Jun 04, 2017 3:55 pm 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 138
I proudly announce the first stable version of BOOTBOOT loader! It's finally done and bugfixed! :-)

Files
Download and source
* mbr.bin - 512 byte stage 1 loader, works as MBR and as a chainloaded VBR
* bootboot.bin - 8192 byte stage 2 loader, can be booted with mbr.bin, a Multiboot compliant loader, or as a BIOS Expansion ROM
* bootboot.efi - UEFI version of the loader, with -? argument gives a little usage help
* bootboot.h - C header file to include in your kernel

How to use
1. create an x86_64 elf kernel (include bootboot.h in it)
2. put it along with other files in an archive. The format doesn't matter at all, just make sure your kernel is the first elf executable inside. Use your own filesystem if you like! Optionally you can compress it with gzip.
3. create a GPT disk image with a small, bootable FAT12/16/32 partition on it (attribute bit 2 set or use ESP's GUID).
4. create a BOOTBOOT directory on that partition, and copy your archive as INITRD there.

Now you can directly boot up your OS from that disk with bootboot.bin in ROM (qemu: -option-rom argument, bochs: optionromimageX config variable), no blessing or setup required at all! :-)

Passing parameters
Simply create a text file holding "key=value" pairs at FS0:\BOOTBOOT\CONFIG. Fill up with newline characters to be exactly 4096 bytes (1 page) long. The loader parses only 3 keys: width and height for framebuffer size, and kernel to specify an elf executable inside initrd (only if you choose to use cpio or tar archives as initrd. For all the other formats the first elf will be loaded). All other keys are up to you and should be parsed by your kernel. Here's an example configuration file for OS/Z.

Machine state on boot
When your elf kernel gains control, the CPU is already in long mode, with IRQs disabled and interrupts turned off. It will be mapped at the highest page directory (-2M...0 if you like). All information gathered in a platform independent structure, like boot time, memory map, framebuffer characteristics, ACPI and SMBI pointers etc. (see bootboot.h). Detailed description and an example kernel with linker script is also provided on github.

BOOTBOOT hides all the tricky and boring details of the early boot process (A20, E820, VBE, LocateProtocol, GOP->QueryMode etc.), so you can focus on writing your kernel.

Tested with
Standard (mbr.bin): qemu 2.9.0, bochs 2.6.9, virtualbox 5.1.22 and real hardware (on USB stick and HDD)
Option ROM: qemu, bochs
UEFI: qemu (OVMF r15214), virtualbox


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Jun 07, 2017 9:45 am 
Offline
Member
Member
User avatar

Joined: Fri Oct 27, 2006 9:42 am
Posts: 961
Location: Athens, GA, USA
OK, I am... puzzled as to why this is a thing. I mean, in a virtual machine, yes, it sounds great, but it rather leaves you stuck once you move to live hardware. I suspect I am missing something, and would appreciate it if you clarified things a bit.

The whole point of an option ROM is that it comes as part of the PCI device it is with; which device's option ROM would you use? The only one that would make sense would be that of the drive you are booting from itself - but most modern SATA drive connectors bypass the PCI bus, and most drives don't have any system-accessible ROM anyway (the on-drive controllers have their own processors, meaning that each drive is, in effect, a co-processor) even if the drive has any flashable ROM ( better-quality ones do, but many cheaper ones just use masked ROM and leave you hanging if there is a bug), you still couldn't use it.

The pending rise of non-volatile memory systems such as 3D XPoint and MRAM will probably quash even that, since eventually the goal is not so much to use it for SSDs, but as part of the main memory. It is very likely that this will put an end to the concept of a separate boot BIOS (in the general sense that covers both PC-BIOS and UEFI) entirely, as things move more towards a persistence-based model in which the OS and session data are resident in non-volatile memory, with the kernel hot-swapped for most updates, and only rebooted in the case of a catastrophic failure - sort of like a return to the days of core memory.

(There's a famous story about the PDP-1 and PDP-6: the factory techs would use Spacewar! as the final shakedown for each new computer, as it was the one non-test program they had that exercised every part of the system. Then, because core was non-volatile, it would be ready to run the next time it was booted, so the installation techs would also then use the game to test the system in the field. =D> )

Anyway, my point is that when and if that becomes commonplace, I would expect that in day-to-day operation the startup sequence would be dominated by something like a CRC performed on the kernel rather than on loading the OS. The system loader itself would probably be just a protected section of NVM which could only be updated with a special permission, and would simply hold the startup code for a kernel (or, more likely, a rump hypervisor that in turn would be able to start one or more kernels and manage switching between them).

So, as I said, I think I must be missing something with all this.

_________________
Rev. First Speaker Schol-R-LEA;2 LCF ELF JAM POEE KoR KCO PPWMTF
μή εἶναι βασιλικήν ἀτραπόν ἐπί γεωμετρίαν
Lisp programmers tend to seem very odd to outsiders, just like anyone else who has had a religious experience they can't quite explain to others.


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Jun 07, 2017 4:41 pm 
Offline
Member
Member
User avatar

Joined: Sat Jan 15, 2005 12:00 am
Posts: 8088
Location: At his keyboard!
Hi,

Schol-R-LEA wrote:
OK, I am... puzzled as to why this is a thing. I mean, in a virtual machine, yes, it sounds great, but it rather leaves you stuck once you move to live hardware. I suspect I am missing something, and would appreciate it if you clarified things a bit.

The whole point of an option ROM is that it comes as part of the PCI device it is with; which device's option ROM would you use?


I'd assume network card ROM would be the most common.

Schol-R-LEA wrote:
The pending rise of non-volatile memory systems such as 3D XPoint and MRAM will probably quash even that, since eventually the goal is not so much to use it for SSDs, but as part of the main memory. It is very likely that this will put an end to the concept of a separate boot BIOS (in the general sense that covers both PC-BIOS and UEFI) entirely, as things move more towards a persistence-based model in which the OS and session data are resident in non-volatile memory, with the kernel hot-swapped for most updates, and only rebooted in the case of a catastrophic failure - sort of like a return to the days of core memory.


The entire concept of "persistent OS in non-volatile system memory" is completely broken and unusable; specifically because there's no way to determine how much data got trashed between the previous "shutdown" and booting it again. To guard against this you need to fence off "safe for persistent data" areas and apply some sort of standardised partitioning convention (and as soon as you do that you end up with "boring old storage device where the only difference is that it just happens to be mapped into the physical address space"). You'd also need some way to say "this data should never be persisted" (for security reasons - e.g. to ensure secure deletion of passwords, temporarily decrypted data, etc; that might remain in memory when power is turned off at exactly the wrong time).

Then there's the fact that, after 60 years of trying (and failing) to fix the "almost all software is full of bugs" problem, the first thing everyone does when things go wrong is reboot to flush out everything that got mangled and restart the system from a clean slate.

Schol-R-LEA wrote:
Anyway, my point is that when and if that becomes commonplace, I would expect that in day-to-day operation the startup sequence would be dominated by something like a CRC performed on the kernel rather than on loading the OS. The system loader itself would probably be just a protected section of NVM which could only be updated with a special permission, and would simply hold the startup code for a kernel (or, more likely, a rump hypervisor that in turn would be able to start one or more kernels and manage switching between them).

So, as I said, I think I must be missing something with all this.


You seem to have forgotten that "non-volatile system memory" doesn't actually exist now (and if it was released tomorrow it'd take at least 10 years before it becomes ubiquitous).



Cheers,

Brendan

_________________
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Thu Jun 08, 2017 10:19 am 
Offline
Member
Member
User avatar

Joined: Thu Oct 13, 2016 4:55 pm
Posts: 138
@Schol-R-LEA: Yes, Brendan is right, most likely it would be a network card's ROM. But not only, you can put that image on the motherboard with flashrom for example. Coreboot supports additional option ROM payloads (should work, but not tested so far).
The point is, it's easy to use in simulators; you can create a hardware to support it directly; but if you don't want to, you can still use it from disk (either with BIOS or with UEFI). You have a lot of options to boot the same long mode environment, and no configuration needed. It's useful when you rapidly changing your kernel or initrd (which is a common case during development). Honestly I don't like GRUB2, it's way to over-complicated to my taste, and it's not able to boot a kernel from my filesystem. Also it loads kernel and initrd as separate files (which is common for a monolithic design), whilst BOOTBOOT loads the kernel from the initrd (more suitable for a microkernel architecture imho).

And about presistent OS, I would really like to see it, but still, it must have an option to do a clear reboot (just to reset the whole system).


Top
 Profile  
 
 Post subject: Re: OS/Z and BOOTBOOT Protocol
PostPosted: Wed Sep 20, 2017 11:51 am 
Offline
User avatar

Joined: Wed Sep 20, 2017 11:10 am
Posts: 2
Dear All,

I've made improvements to the BOOTBOOT loader. First of all, to honor osdevers on the Windows platform, BOOTBOOT now supports PE kernels (only 64 bit PE32+ format) along with ELF64. I do not have a Windows machine, so I've used objconv to create PE executable which may or may not differ to the one produced by VS. If somebody would be nice to compile the example kernel and attach the resulting PE image, I could test with that.

Also because it's popular these days, I've added Simple File System support. Both 1.0 (Brendan's version as discribed on the wiki) and 1.1 (BenLunt's version as described in his pdf). Since there's no utility or source code to generate such an image, it's a blind implementation totally untested. Once BenLunt finishes with his tool, and I can compile it under Linux, I'll gladly use it for testing and debugging the SFS 1.1 support. I've implemented it in C for UEFI, as well as in Assembly for the BIOS/Multiboot version. As far as I know, this makes BOOTBOOT the first boot loader that actually supports SFS (either in initrd file or on a partition).

Another good news, although it's not ready to be published yet, but the Raspberry Pi 3 version is promising :-) With that BOOTBOOT will support the same boot protocol on multiple platforms (meaning common memory map format everywhere, compatible framebuffer setup and common way to pass environment parameters to kernels accross architectures). I have a question for you though. Currently I don't use this feature, but I set the registers with values before I pass control to the kernel (a magic, and linear addresses of structures). Removing that feature would make the protocol more universal across platforms, but if somebody says "hey, I surely could use them in my assembly kernel!", I'll leave the registers in the spec.

And last thing, a minor change, I've renamed mbr.bin to boot.bin, because it caused trouble for some people to understand it's a boot sector code, which can be used in MBR and in VBR too. Hope now it's obvious. I've updated the documentation accordingly.

Best wishes,
bzt


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group