Qemu + EFI + Grub2: error in ExitBootServices

[vagran]

Hi,
I have recently started to migrate my OS to EFI. I am using the latest Qemu (0.14.0) with TianoCore EFI published on Qemu site. OS loader is GRUB2 (1.98). The problem is that GRUB fails to ExitBootServices EFI function - it stops for a while in this function and system is rebooted after that. Commenting out the call to this function in GRUB resolves the problem (OS is loaded) but it is not acceptable workaround for me because I plan to use EFI services during my kernel initialization so it should have proper state. Does anybody encountered similar problem?
And more general question: does anybody have experience with EFI support implementation? Any comments, advices, recommendations?

[vagran]

The problem was solved. Actually it was my fault. I have used traces for grub debugging and placed grub_printf() call after ExitBootServices() called. It caused this erroneous behavior since GRUB uses EFI for console communications.

[Brendan]

Hi,

The problem was solved. Actually it was my fault. I have used traces for grub debugging and placed grub_printf() call after ExitBootServices() called. It caused this erroneous behavior since GRUB uses EFI for console communications.

That's good news!

I'm wondering...

Would you mind describing how to work with EFI? Not the stuff that's included in the EFI/UEFI specifications, but things like how to setup tools, etc?

Ideally, what I'd like to do is put together a "Working with EFI/UEFI" tutorial on the Wiki, that shows a very simple "Hello World" application, explains how to setup GCC (and LD or objcopy, etc) to generate a binary, how to access the EFI API (where to get the header files?), how to get the resulting binary onto something like a bootable USB flash device, etc.


Thanks,

Brendan

[vagran]

Actually my objective is not to generate EFI binaries/applications but to make OS running in EFI environment. The only EFI binary required for me is multiboot-compliant loader such as GRUB2. The next step would be accessing EFI services from OS kernel to get memory maps, MP tables, ACPI tables etc - all the stuff which used to be retrieved from BIOS. I'm not yet very competent how to do this because I just started feasibility study for this task. Currently I have working Qemu with EFI BIOS and EFI GRUB2 which is able to load and start my old kernel. So compiling for EFI and "Hello world" running from EFI shell is a little bit out of scope of my project. But I strongly agree that this topic should represented in details somewhere. I would gladly share all my findings and experience in this area.

If you are interested in details of what is achieved in my case now, here they are:

* Qemu: Nothing special has been yet done - just compile latest version for required architecture with required options. When launching specify with "-bios" parameter the EFI firmware image available from http://wiki.qemu.org/Download/.
There is also requirement for disk image. It should contain so called ESP - EFI System Partition which can be formatted only in FAT filesystem. Partitioning could be either MBR or GPT (GUID partition table). I am using the second one. It can be created, for example, in such way:

Code: Select all

# 32MB raw disk image
dd if=/dev/zero of=efi.disk bs=1024 count=32768
# gdisk utility used for creating GPT from http://sourceforge.net/projects/gptfdisk/
gdisk efi.disk
# use "o" to create new GPT, "n" to add partition with "ef00" type, "w" to write it in the file
# Check start sector number of your system partition by "p" command or with "gdisk -l efi.disk", e.g.:
#Number  Start (sector)    End (sector)  Size       Code  Name
#   1            2048           65502   31.0 MiB    EF00  EFI System
# mount partition to loopback device
sudo losetup -o $((2048*512)) /dev/loop0 efi.disk
# create FAT12 filesystem, for larger disks it could FAT16 or FAT32
sudo mkdosfs -F 12 /dev/loop0
# mount new partition to some directory and fill it with your files
mkdir mnt
sudo mount /dev/loop0 mnt
....
# unmount when done
sudo umount mnt
sudo losetup -d /dev/loop0

You can create "startup.nsh" file in the root of your ESP. It is interpreted as EFI shell script and executed at startup if present. I have following line there to boot GRUB:

Code: Select all

fs0:\efi\grub\grub.efi

Actually I am using Perl script to create system image. I have attached it to the post.

* GRUB: I have configured GRUB 1.98 with the following command:

Code: Select all

../src/configure --prefix=$PREFIX --with-platform=efi --target=i686 --disable-nls --enable-efiemu=no

Qemu EFI currently can launch only 32-bit binaries so you need to compile GRUB with target i686 even if you plan to emulate 64-bit system.
After compilation you can build your GRUB2 EFI binary specifying list of modules to build-in:

Code: Select all

grub-mkimage -o grub.efi boot linux part_gpt fat ext2 normal sh configfile lspci ls reboot datetime loadenv search lvm help multiboot elf

Note that I am using "part_gpt" module because I use GPT. I have spent some time to figure out why GRUB cannot find its modules and configuration file on disk image which is provided with TianoCore EFI. The reason was that it had MBR partitions table so if you still use it you should add "part_msdos" module.
You should copy your GRUB binary, its modules and configuration file on your disk image:

Code: Select all

sudo mkdir -p mnt/efi/grub
cp grub.efi $YOUR_PREFIX/lib/grub/i386-efi/*.mod $YOUR_PREFIX/lib/grub/i386-efi/*.lst mnt/efi/grub/
# create mnt/efi/grub/grub.cfg if necessary

"multiboot" command in GRUB could boot your multiboot-compliant kernel:

Code: Select all

search --file --set /boot/kernel
menuentry "YourOS" {
	multiboot /boot/kernel --root ramdisk0
}

I have also tried to found why some delay occurs when it loads my kernel and found that disk cache blocks size is set to too small value. I have increased it up to 64k and it started to load the kernel almost instantly. Increasing this value more gives no effect.

Code: Select all

--- include/grub/disk.h.old     2010-03-06 22:51:37.000000000 +0200
+++ include/grub/disk.h 2011-02-24 08:39:37.893224987 +0200
@@ -135,8 +135,8 @@
 #define GRUB_DISK_CACHE_NUM    1021
 
 /* The size of a disk cache in sector units.  */
-#define GRUB_DISK_CACHE_SIZE   8
-#define GRUB_DISK_CACHE_BITS   3
+#define GRUB_DISK_CACHE_SIZE   128
+#define GRUB_DISK_CACHE_BITS   7
 
 /* This is called from the memory manager.  */
 void grub_disk_cache_invalidate_all (void);

As I already said I'm not interested in creating applications for EFI so I did not perform any study in this direction. As far as I know it should not be very difficult - GNU binutils could be compiled for efi-app-ia32 or efi-app-x86_64 target. Also there is some gnu-efi package with some useful tools available there: http://sourceforge.net/projects/gnu-efi/.

Currently it is all my progress in this area because now I have temporarily switched to problem with GDB + Qemu (I have also started migration to 64 bits architecture and there is a problem in the latest Qemu - when it is emulating x86_64 it always sends 64-bits registers to remote GDB, even if CPU is currently in 32-bit mode. I have initiated discussion with Qemu developers which could probably lead to patch (at least for me I will definitely have a private patch)) so studying in this area from my side is frozen for a little. Probably some time will be spent before I will return to EFI topic because for now I need to redesign my kernel linkage to make it compilable by x86_64 gcc and have 32-bit entry point. After that I will continue investigation about accessing EFI services from my kernel. If someone is interested in this story continuation I could post new details either here or on some wiki page.

[Brendan]

Hi,

Actually my objective is not to generate EFI binaries/applications but to make OS running in EFI environment.

My intention is to create an "EFI application" to begin with, as the only difference between an application and a boot loader is that the boot loader doesn't return (and eventually calls "ExitBootServices()"). It's exactly the same executable file format and API.

I spent most of yesterday researching, etc. I ended up with VirtualBox setup to boot into an EFI shell, and an ISO9660 CD with a 32-bit "hello world" EFI application on it. The EFI shell refuses to start the application. I also tried some utilities from rEFIt and found that the problem is my application, and not the EFI shell or the CD image.

I also don't think it's my "hello world" code. If that was wrong in any way I'd expect it to crash or something. I think it's the way I'm converting the code into an EFI application that causes the EFI shell to think it's not an executable (e.g. messed up PE32+ headers or something).

Late last night I followed your link (http://sourceforge.net/projects/gnu-efi/) and read some of the notes in the gnu-efi source. I think there's large amounts of "quirks" I'm missing, and need to spend more time researching them.

The notes in the gnu-efi source (and some of the comments I saw in binutils source code when I was trying to figure out why "efi-app" formats weren't listed as supported) give me the impression that support for EFI applications in binutils could be an after-thought rather than something well designed. I'm also tempted to wonder if some of the problems (e.g. "some loaders can't handle uninitialised data sections properly") might be caused by subtle differences between the headers, etc that Microsoft's tools would produce and the headers, etc that binutils produce. Finding (and testing) other ways of generating EFI applications that don't involve binutils might be worthwhile. So far the only other choices I've found seem to be Microsoft's tools and FASM (and FASM should probably go at the top of my "things to research" list).

The only EFI binary required for me is multiboot-compliant loader such as GRUB2. The next step would be accessing EFI services from OS kernel to get memory maps, MP tables, ACPI tables etc - all the stuff which used to be retrieved from BIOS.

When EFI starts a program it passes 2 parameters to the program's "main()" function. One of those parameters is the address of the "EFI System Table", which is the top of a tree of structures that contain everything including handles, pointers to ACPI and SMBIOS tables, function pointers for all the EFI functions (including the run-time functions), etc. If GRUB doesn't tell you the address of the "EFI System Table", then you're mostly screwed - you can't reliably get anything that multiboot doesn't give you (and multiboot doesn't give you much more than a memory map - no ACPI tables, no SMBIOS tables, etc).


Cheers,

Brendan

[vagran]

Hi,
When EFI starts a program it passes 2 parameters to the program's "main()" function. One of those parameters is the address of the "EFI System Table", which is the top of a tree of structures that contain everything including handles, pointers to ACPI and SMBIOS tables, function pointers for all the EFI functions (including the run-time functions), etc. If GRUB doesn't tell you the address of the "EFI System Table", then you're mostly screwed - you can't reliably get anything that multiboot doesn't give you (and multiboot doesn't give you much more than a memory map - no ACPI tables, no SMBIOS tables, etc).

Actually I have not yet researched this question. Do you mean that in EFI environment some existing multiboot-complient loader is useless for an OS because it doesn't pass required information in the OS kernel? It a little bit confuses me. The first question which comes to my mind is how other EFI-compatible OSes are working. Does it mean that you need either to write your own loader or make your kernel appear as EFI application for an EFI and load it directly? The second way doesn't seem very nice - EFI doesn't support ELF format but making whole the kernel in PE is a hell. May be there is some way to make GRUB pass required pointers to the kernel? May be there are any updates in multiboot specification regarding EFI? Probably Linux kernel code can be inspected to figure out how they do it (and they somehow do it - Linux is booted by GRUB on EFI machines).
These are just my thoughts when I am thinking in which direction to move forward with this question.

[Brendan]

Hi,

When EFI starts a program it passes 2 parameters to the program's "main()" function. One of those parameters is the address of the "EFI System Table", which is the top of a tree of structures that contain everything including handles, pointers to ACPI and SMBIOS tables, function pointers for all the EFI functions (including the run-time functions), etc. If GRUB doesn't tell you the address of the "EFI System Table", then you're mostly screwed - you can't reliably get anything that multiboot doesn't give you (and multiboot doesn't give you much more than a memory map - no ACPI tables, no SMBIOS tables, etc).

Actually I have not yet researched this question. Do you mean that in EFI environment some existing multiboot-complient loader is useless for an OS because it doesn't pass required information in the OS kernel?

As far as I can tell, the basic idea of multi-boot is to provide an abstraction, such that the OS never needs to care about the firmware itself. In my opinion, this is the right way to do it - it would mean any (multiboot) OS could work on any computer (regardless of whether the firmware is PC BIOS or EFI/UFI or whatever else) with a lot less hassle.

However, (also in my opinion) while the general idea is right, the implementation is lacking. For example, multi-boot doesn't tell you where the ACPI tables, MPS tables and SMBIOS tables (if present) are. OS's designed for "PC BIOS" work around this by scanning certain areas looking for these tables, which seems fine until you start looking at EFI where you can't do that reliably - "Scan the first 1 KiB of the EBDA looking for <foo>" doesn't make any sense when there's no EBDA.

For the abstraction to be complete (to make sure no OS will ever need to care what the underlying firmware is/was), the abstraction has to cover everything any OS could want, on any supported platform (e.g. including ARM, MIPS, Itanium, whatever). This means lots of stuff are missing from the "multi-boot abstraction" - how to determine the PCI configuration space access mechanism, how to get/set the time and date, support for trusted systems (authenticated boot), some standardised way an OS can modify it's own boot configuration (e.g. setting "multiboot variables" during boot maybe), a way to get standardised device lists on embedded systems (where there's no other way - similar to the "device tree" that uboot provides on these systems), etc.

After all of those problems are fixed and multiboot becomes a complete abstraction; then you've basically got "yet another abstraction". It'd be just as easy to port an existing abstraction (like EFI or OpenFirmware or whatever) and use that everywhere instead of inventing something new and different.

It a little bit confuses me. The first question which comes to my mind is how other EFI-compatible OSes are working. Does it mean that you need either to write your own loader or make your kernel appear as EFI application for an EFI and load it directly?

Most OSs that support EFI have a boot loader (an EFI application) that loads the kernel (and whatever other files) into RAM; but (at least in theory) there's no real reason you couldn't design the kernel as an EFI application instead.

May be there are any updates in multiboot specification regarding EFI?

Maybe - "Multiboot 2" is only a draft specification, and nothing prevents the GRUB developers from changing it (and even breaking compatibility) whenever they like.

Probably Linux kernel code can be inspected to figure out how they do it (and they somehow do it - Linux is booted by GRUB on EFI machines).

Linux does not support multi-boot. Linux has it's own requirements for the hand-off between the boot loader and the kernel; and GRUB supports that. A lot of well known OSs are the same and don't support multiboot either. NetBSD is a little different in that they have optional multi-boot support (but still has it's own native boot method). There are a few kernels that do rely on multiboot - L4, GNU Hurd and newer versions of Solaris/OpenSolaris. Of course all of this is probably the original version of multi-boot on PC BIOS, and not "multi-boot 2" or EFI. I'm not too sure if there are any OSs that support "multi-boot 2" and EFI (on x86) yet.


Cheers,

Brendan

[vagran]

Yes, seems you are right. I just had an opinion that multiboot could be some kind of standardized way of loading OS kernel. But seems in case of EFI the most preferable approach is to create EFI binary which could be loaded by EFI built-in bootmanager. Probably I will try to do it. Current idea is to create EFI binary which will contain gzipped ELF kernel in its data section, extract it in required location and pass EFI arguments to it. Firstly I will try to do it with binutils. Also I have noted another option - GRUB2 has mkimage utility. If you look in its internals (util/i386/efi/grub-mkimage.c) you'll see that it just converts ELF relocatable image to EFI PE. It could be a fallback option if nothing can be done with binutils.
BTW do you know how to preserve Qemu NVRAM between reboots? EFI stores there its variables, e.g. bootmanager menu, so it would be very useful.

[Brendan]

Hi,

Yes, seems you are right. I just had an opinion that multiboot could be some kind of standardized way of loading OS kernel.

Don't get me wrong - "multi-boot 2" could become some kind of standardized way of loading OS, eventually, when it's no longer a "draft" and all the issues are fixed. Unfortunately the GRUB project don't seem to work like that - they don't design a draft specification, get feedback, make any changes necessary, etc, and then release a final specification before starting work on an implementation of that specification. Instead they start work on an implementation, then update the draft specification to match their implementation, so that nobody can tell what they're going to end up with until they've moved on to "GRUB 3".

But seems in case of EFI the most preferable approach is to create EFI binary which could be loaded by EFI built-in bootmanager.

I'm not too sure here either. I'd assume there's some problems in some tools (e.g. binutils) and some problems in current implementations of EFI firmware. I've been thinking of using NASM's "flat binary" format with a PE32+ header defined with macros to get around any toolchain issues (and partly because I'd like to keep the number of tools required to build my project down to a minimum); and then using tricks to write purely position independent code with a single ".text" section and nothing else to get around any bugs in the PE32+ loader in EFI firmware. This initial loader would load/unpack a "stage 2" (which would be in my own executable file format and not PE32+) and setup sections properly, and provide a simplified interface to EFI functions (so that the same "stage 2" can be used on both 32-bit EFI and 64-bit EFI).

I still need to do plenty of research though, and there's other issues I need to sort out (integrating the "EFI dependant" boot stages with the later stages of my boot code).

Also I have noted another option - GRUB2 has mkimage utility. If you look in its internals (util/i386/efi/grub-mkimage.c) you'll see that it just converts ELF relocatable image to EFI PE.

I wouldn't be surprised if that is just a wrapper around the "gnu-efi" package and binutil's "objcopy".

BTW do you know how to preserve Qemu NVRAM between reboots? EFI stores there its variables, e.g. bootmanager menu, so it would be very useful.

Sorry - I have the same problem with VirtualBox and haven't found a work-around. I have a feeling that (on real hardware) NVRAM is actually a reserved area within the firmware's flash ROM, and that most emulators don't emulate flash ROM.


Cheers,

Brendan

[Brendan]

Hi,

Just a quick update - I've finally managed to get a "do nothing" EFI application to work...

I used FASM to create the 32-bit EFI application from this source code:

Code: Select all

format pe dll efi
entry main

section '.text' code executable readable

main:
    mov eax, 0
    retn

section '.data' data readable writeable
section '.reloc' fixups data discardable

That's all you need to get started - no linker scripts, no strange cludges, no ugly hacks. I've spent days trying (and failing) to get binutils to work, and it seems FASM did it correctly "out of the box" with no hassle whatsoever.

Therefore my advice is to use FASM and forget about binutils.

Of course I reserve the right to ignore my own advice - the first thing I'm going to do is use the "known good" PE32+ headers, etc generated by FASM as the basis of a "flat binary" EFI application written in NASM...

The worst part is that I should've been doing exam revision, and won't be able to do much for the next 4 days or so.


Cheers,

Brendan

[vagran]

I also have an update. I tried to build EFI application using gnu-efi and binutils and discovered that it works without any problems. So if you want to use GNU toolchain for building an EFI application

• download gnu-efi from http://sourceforge.net/projects/gnu-efi/
• ensure your binutils support efi-app-ia32 target (it is displayed as "pei-i386" in "--help" output in the latest binutils). I have added "--enable-targets=i386-efi-pe" option when configured binutils-2.21.
• execute "make" command in gnu-efi root directory. Several sample applications will be built in "apps" directory.

You can examine its build log to understand what's going on and integrate required linker script and make rules into your project. Actually it builds several run-time support libraries with all sources built with -fpic. Also it compiles assembler file with entry point code and dummy ".reloc" section which just points to EFI loader that it is relocatable PE. Real ELF relocations are preserved (they are saved in data section) and used at startup to relocate the binary to real load address by its own relocation function. All object files are linked into an ELF shared library using specific linker script which defines entry point, "ImageBase" symbol used for image relocation, specific sections ordering and alignment. Resulting ELF shared library is converted to EFI PE by providing "--target=efi-app-ia32" option and explicit list of sections to process.

So it is not a problem to use binutils too. Personally for me it is preferable exactly by the reason you have mentioned - in order to minimize number of build tools in the project toolchain. Currently I have only GCC and binutils and it seems to be enough for all kinds of binaries.

Seems we have no open questions left in this topic. I'm going to proceed to loader implementation - gzip my kernel, convert it to object file by objcopy, use linker script to link it to resulting binary defining symbols for address and size where it is stored, and decompress and load it to required addresses in run-time using zlib and libelf. Also I will probably try to enhance Qemu to preserve EFI NVRAM content (the problem is that I cannot find sources of TianoCore ported version which is used on Qemu).

[vagran]

Continuing the topic...
I have found that on the latest Qemu release (0.14.0) and the latest development snapshot EFI BIOS doesn't properly initialize VGA and serial console so you will not see any output from it. Seems the problem is in EFI BIOS itself. I have initiated discussion and provided workaround in Qemu:
http://lists.nongnu.org/archive/html/qe ... 00186.html

The following patch could be used as workaround in Qemu until the EFI BIOS will be fixed:

Code: Select all

diff --git a/hw/pci.c b/hw/pci.c
index 8b76cea..bcf9b16 100644
--- a/hw/pci.c
+++ b/hw/pci.c
@@ -163,8 +163,9 @@ void pci_device_reset(PCIDevice *dev)
    pci_device_deassert_intx(dev);
    /* Clear all writeable bits */
    pci_word_test_and_clear_mask(dev->config + PCI_COMMAND,
-                                 pci_get_word(dev->wmask + PCI_COMMAND) |
-                                 pci_get_word(dev->w1cmask + PCI_COMMAND));
+                                 (pci_get_word(dev->wmask + PCI_COMMAND) |
+                                 pci_get_word(dev->w1cmask + PCI_COMMAND)) &
+                                 ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY));
    pci_word_test_and_clear_mask(dev->config + PCI_STATUS,
                                 pci_get_word(dev->wmask + PCI_STATUS) |
                                 pci_get_word(dev->w1cmask + PCI_STATUS));

[Brendan]

Hi,

Today I had some time, and continued messing with EFI.

I found out previously that you can generate all the headers, etc with NASM, except for one problem - the checksum in the "optional" PE header. To get around that I slapped together a utility in C to generate and insert the necessary checksum. That seemed a bit messy to me (doing half in NASM and half in a C utility) so today I modified that utility to generate all the headers, etc.

Now I can generate a pure flat binary in NASM, and use my utility to convert that into a (32-bit) EFI application. I still need to tweak it a little (see how much I can omit before EFI refuses to execute it), but the actual code must be on a 512-byte boundary within the file and the headers are less than that, so I can't trim down the file size at all. I also need to look into generating 64-bit EFI applications (which probably isn't very different, other than a different value in a "machine type" field).

As far as relocations go; for 64-bit EFI applications I'll just use RIP relative addressing and won't need any relocations. For 32-bit EFI applications I'm planning to use my own GDT and create code and data descriptors with "base address = load address", so that I won't need to bother with any relocations for that either (but will need to save and restore the firmware's GDT and change some segments when calling the EFI API).

Tomorrow I'm planning to do the GDT tricks and get a "hello world" working. After that, it shouldn't be too hard to get a boot loader done.

Hopefully I'll get "Qemu + EFI" setup soon too - I can't test 64-bit EFI applications in VirtualBox.

One last thing I wanted to mention - I stumbled across this introduction to EFI programming. There's a few useful tips on there that are worth looking at.


Thanks,

Brendan

[vagran]

I have finally get it working. EFI firmware binary taken from http://sourceforge.net/apps/mediawiki/t ... title=OVMF works fine on the latest Qemu. There is also no longer the issue with EFI NVRAM - now it is stored in a file on EFI System Partition so you can use EFI boot manager and any NVRAM variables required for your loader - they will be preserved between reboots.

[turdus]

I've compiled a working hello world with fasm. I've used the tutorial here (scroll down to "UEFI Programming with FASM" section): http://x86asm.net/articles/uefi-program ... rst-steps/

It works fine on my MacBook which has a 32 bit EFI. Unfortunatelly I had to use linux to compile it, since my macho fasm is out of date. Now I have to upgrade it (compile newest fasm with libc support, convert it with objconv, and link with gcc) to go further.

And I have to find a way to make my kernel BIOS/EFI independent. There's only one part that rely on BIOS, when the crash handler switches to text video mode. That won't work on pure EFI machines (without BIOS compatibility layer which the MacBook has), propably cause a hang or reboot or something even nastier...