UEFI question

[bzt]

The function pointer you pass to StartupThisAP() must be to a function that returns. Said function must return before you call ExitBootServices().

Where did you get that? Both the UEFI PI and the EDK header states that StartupThisAP() can be executed in non-blocking mode, when the BSP is not supposed to wait for the function to return.

If WaitEvent is NULL, execution is in blocking mode. The BSP waits until the AP finishes or TimeoutInMicroSecondss expires. Otherwise, execution is in non-blocking mode. BSP proceeds to the next task without waiting for the AP.

There's absolutely nothing about the function must return before ExitBootServices (it just states that MP Services cannot be called after ExitBootServices). UEFI PI section "Non-Blocking Execution Support" on page 2-191 mentions nothing about the function returning. It only mentions using SingalEvent when timeout occurs, that's all (however it also mentions that zero timeout means infinity, aka no signal).

Using INIT and STARTUP IPIs after calling ExitBootServices().

The whole point of MP Services is to provide a platform independent way of initializing the APs, from MpService.h:

Code: Select all

  MP Services Protocol is hardware-independent. Most of the logic of this protocol
  is architecturally neutral. It abstracts the multi-processor environment and
  status of processors, and provides interfaces to retrieve information, maintain,
  and dispatch.

INIT and STARTUP IPI are not platform independent, and if a loader must use those, then what's the point of MP Services anyway? Is this yet another epic failure on the Intel engineers part?

Another question, STARTUP IPI requires a code block under 1M for real mode startup. How can you allocate a memory for that, when UEFI is managing the memory? How can you tell AllocPages (or any other UEFI memory allocation function) that you need a page that's below 1M? How can you be sure that there's any free memory left below 1M at all? This makes no sense. From the doc:

Caution: Although the Allocate services allow for specific memory allocation, do not allocate specific addresses in a UEFI driver. Allocating buffers at a specific address could result in errors, including a catastrophic failure on some platforms.

Cheers,
bzt

[zaval]

Another question, STARTUP IPI requires a code block under 1M for real mode startup. How can you allocate a memory for that, when UEFI is managing the memory? How can you tell AllocPages (or any other UEFI memory allocation function) that you need a page that's below 1M? How can you be sure that there's any free memory left below 1M at all? This makes no sense. From the doc:

Caution: Although the Allocate services allow for specific memory allocation, do not allocate specific addresses in a UEFI driver. Allocating buffers at a specific address could result in errors, including a catastrophic failure on some platforms.

Are you writing a UEFI driver? That guide is for driver writers, there is no such a limitation for an OSL - to ask allocation functions about specific address range (or even address), consult the UEFI specification. Of course, the requested address might be not available, but if you decided to spin up secondary CPUs in the OSL (yet - in the UEFI Boot Services stage), it's your problem.

As of the original problem, if your clear_screen() thing isn't calling anything, then the answer why ExitBootServices() hangs on that particular machine could be anything - who knows. it's a bug. either in your OSL or UEFI of the machine. Could you avoid touching secondary processors? If so, try it, because it looks like the most suspicious culprit.

[Octocontrabass]

There's absolutely nothing about the function must return before ExitBootServices (it just states that MP Services cannot be called after ExitBootServices). UEFI PI section "Non-Blocking Execution Support" on page 2-191 mentions nothing about the function returning. It only mentions using SingalEvent when timeout occurs, that's all (however it also mentions that zero timeout means infinity, aka no signal).

UEFI PI section "EFI_MP_SERVICES_PROTOCOL" on page 2-181 explains it.

Modules that use this protocol must guarantee that all non-blocking mode requests on all APs have been completed before the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES is signaled.

Is this yet another epic failure on the Intel engineers part?

Yes!

Another question, STARTUP IPI requires a code block under 1M for real mode startup. How can you allocate a memory for that, when UEFI is managing the memory?

UEFI isn't managing the memory after ExitBootServices().

How can you tell AllocPages (or any other UEFI memory allocation function) that you need a page that's below 1M?

I'd try AllocateMaxAddress if I really needed to set up the trampoline before calling ExitBootServices().

How can you be sure that there's any free memory left below 1M at all?

Linux never seems to have any trouble.

Caution: Although the Allocate services allow for specific memory allocation, do not allocate specific addresses in a UEFI driver. Allocating buffers at a specific address could result in errors, including a catastrophic failure on some platforms.

Those guidelines are for UEFI drivers. I don't think a bootloader for one specific platform needs to worry that it won't work on other platforms.

[bzt]

Is this yet another epic failure on the Intel engineers part?

Yes!

We should put their hands in gold (so that they can never ever use a keyboard again). That's just sad. Who gave those guys a diploma anyway...?

UEFI isn't managing the memory after ExitBootServices().

But it does not release memory for run time services either. UEFI drivers will continue to use memory allocated as run time data.

How can you be sure that there's any free memory left below 1M at all?

Linux never seems to have any trouble.

Linux is extremely poorly written in this regard. Linux's SMP code isn't UEFI specific, and we know that under BIOS the < 640k memory is indeed free (not fully used by EBDA for sure). Btw, Linux uses a hardcoded value as default, and it does not check if it could really find a free RAM entry below 640k to replace the address in "bios_start" variable. If an UEFI driver allocates all memory below 0x413 << 10 then a crash with Linux is guaranteed. Not just for the AP trampoline code (set here), but for the uncompression too...

Those guidelines are for UEFI drivers. I don't think a bootloader for one specific platform needs to worry that it won't work on other platforms.

You are missing the whole point. UEFI drivers are running along with your bootloader, and they are started before your code! Therefore any UEFI driver could "steal" the low memory from you.

Unlike BIOS, UEFI does not guarantee that any memory below 1M going to be usable, which is a big problem for SMP. Yes, you could assume that a certain address is going to be fine, but that will result in exactly the same runs-on-many-except-some-machines problem just like with MP Services

Cheers,
bzt

[zaval]

bzt, UEFI is CPU independent and that's why it does not guarantee any weird assumptions about particular address range availability. if you are so inclined to start secondary processors in your OSL, then at least do it after you get control over the address map, since it's needed if you want particular address range availabilty on any machine. there is no way, that all lower 1MB of RAM would be occupied with Rintime Services typed memory. but you already know the right answer - the problem is in your strange decision, not in the FW. why you can't start secondary processors from inside your kernel initialization?

[nullplan]

Linux is extremely poorly written in this regard. Linux's SMP code isn't UEFI specific, and we know that under BIOS the < 640k memory is indeed free (not fully used by EBDA for sure). Btw, Linux uses a hardcoded value as default, and it does not check if it could really find a free RAM entry below 640k to replace the address in "bios_start" variable. If an UEFI driver allocates all memory below 0x413 << 10 then a crash with Linux is guaranteed. Not just for the AP trampoline code (set here), but for the uncompression too...

On x86, SMP is not specific to the firmware. By the way, thank the Intel engineers for not f'ing that up, at least. Other architectures are not so lucky. Today I read the Linux code for starting secondary processors on PowerPC. It calls OpenFirmware. There is so little of a standard of how to do it there that even Linux, which sometimes resembles "NIH the OS" prefers to just call the firmware to do it.

Anyway, the address given is 0x104c00. Pardon my academic elitism, but 6 digits is more than 5 digits, so this is more than 1MB. If a driver did as you proposed, it would allocate all of low memory. I would wager Windows wouldn't work very well with wonky allocations like that, either. Dammit, ran out of w-words. Low memory is simply required for SMP startup on x86, since any AP starts in 16-bit real mode and cannot access any memory beyond 1MB. Plus, the INIT IPI always specifies a start address in low memory.

Unlike BIOS, UEFI does not guarantee that any memory below 1M going to be usable,

Any such system will be unable to support SMP with any major operating system, and their customers are going to be very upset at such a development. Especially since both Intel and AMD are pushing more and more ridiculous numbers of cores and threads, because apparently we have reached peak single core performance. Having a system have not even a single page of memory free in low memory is simply not workable on x86. UEFI might not guarantee this, but the facts of the matter do.

[bzt]

On x86, SMP is not specific to the firmware.

Yes, it is. SMP initialization code is very very architecture, platform, and sometimes firmware specific. On x86 you have at least three firmware, BIOS/MP tables, BIOS/ACPI and UEFI/ACPI at least. But for example, with old RaspberryPi firmware, you would have to issue a Mailbox call to awake APs. Then they've changed the firmware to start all APs at once. And with the latest firmware, only BSP runs and now you can release the spinlocks with direct memory writes without touching the firmware.

According to the UEFI PI spec, the whole purpose of MP Services supposed to be hiding all of that and provide a platform independent, common interface for SMP in the firmware.

By the way, thank the Intel engineers for not f'ing that up, at least.

What good is MP Services from the OS perspective if the kernel can't use it? It needs all INIT IPI + STARTUP IPIs and real mode and prot mode trampoline implemented in UEFI. Then all that architecture specific sh*t has to implemented *again* in a kernel later when the required memory might be not free, just because APs must be stopped on ExitBootServices? That's just stupid. Why shouldn't a kernel use MP Services in the first place? (Which works on many boards, BTW)

With MP Services, the initialization (and protmode/longmode trampoline) happens *before* any UEFI driver started (in CpuPkg to be precise), therefore any driver can freely use low mem. Then nobody cares what memory the UEFI drivers are allocating, because the APs are already started up and awaiting in native mode. The StartupThisAP method does not actually start up the AP, it just dispatches a native mode function on the AP. (And Hell, even the method name suggests it's for starting up the APs and not do-something-boot-time-only-stuff!!!)

This way MP Services interface is really and truly architecture independent, but what good does that make if you can't use it from your OS?

Anyway, the address given is 0x104c00. Pardon my academic elitism, but 6 digits is more than 5 digits, so this is more than 1MB.

That *0x413 << 10 address was copy'n'pasted right from the Linux source. That's why I wrote if the loop in lines 73 - 104 fails to modify the "bios_start" variable, then a crash is guaranteed. BTW, the value of 0x413 is NOT checked in any way either... It could be right in the middle of an UEFI driver allocated page!

Unlike BIOS, UEFI does not guarantee that any memory below 1M going to be usable,

Any such system will be unable to support SMP with any major operating system

Yeah, because Intel engineers messed this up too. They could easily boot SMP if MP Services were just simply run time services... Or if APs weren't expected to be stopped on ExitBootServices call (many firmware allows that, but it should be in the spec).

Cheers,
bzt

[zaval]

All protocols are Boot Service time things and cannot be Runtime Service time things. Drivers and the FW core itself, implementing them, uses Protocol management Boot Services during protocols operation, among other things, belonging to Boot Services. None protocol was intended for usage by an OS, it's all for booting OSs. and honestly, that MP stuff is just a show off thingy. The need of redoing SMP initialization by the OS is not a flaw of the FW, just like resetting the GPU isn't, because the FW does that for its own purposes and what it has done or hasn't, shouldn't be a concern for an OS, since the latter still needs to do way much more and often - the other way, because FW initializes something just as much as needed for loading the OS, the OS does in full. MP isn't needed for OS loading, it was put there probably because of the "oh, look, let's make it SMP, it would be so rad" reason.

[nullplan]

On x86 you have at least three firmware, BIOS/MP tables, BIOS/ACPI and UEFI/ACPI at least.

MP tables are of interest only to historians and those OS developers too lazy to integrate ACPICA or similar. I for one do not plan to support multi-processor 486 boards any time soon, so MP tables are about as relevant to me as the river level of the Amazon from last year. And the only difference between BIOS/ACPI and UEFI/ACPI is how to find the XSDT.

RaspberryPi

Yeah, I did say x86, didn't I? On all other architectures, these things are very flexible, as I tried to point out with the PowerPC example.

According to the UEFI PI spec, the whole purpose of MP Services supposed to be hiding all of that and provide a platform independent, common interface for SMP in the firmware.

Yup, and then they added a requirement that the APs return to the firmware before calling ExitBootServices(), thereby rendering the whole thing useless. It's almost a good system. I have come to realize that "almost" is just a polite way of saying "not".

Looking at the UEFI membership list, I wonder if this is yet another case of diffusion of responsibility. Can you tell me who broke the spec? No, this way it could have been anyone. And you have no-one to blame.

What good is MP Services from the OS perspective if the kernel can't use it?

No good, obviously. Apparently they needn't have bothered.

That *0x413 << 10 address was copy'n'pasted right from the Linux source.

I misread the code. I thought bios_start was assigned an address of 0x413 << 10. No. It is assigned the value at address 0x413, times 1024. Apparently, the BIOS data area contains the RAM size in kB at that location. Of course, Linux assumes the BDA is present unless UEFI is present. The search loop would work great if it could return failure. However, if that particular loop can't find anything, your system is broken anyway, since the kernel cannot be decompressed, then. If they wanted to change that, they'd have to change the architecture of the code.

[bzt]

All protocols are Boot Service time things and cannot be Runtime Service time things.

But they could be! For example, you could switch the screen resolution from your OS any time using BIOS/VBE. But you can't do that with GOP (being a boot time service). There's absolutely no technical nor hardware related reasons why those protocols are inaccessible for the OS, except shitty UEFI design and implementations.

MP tables are of interest only to historians and those OS developers too lazy to integrate ACPICA or similar. I for one do not plan to support multi-processor 486 boards any time soon, so MP tables are about as relevant to me as the river level of the Amazon from last year. And the only difference between BIOS/ACPI and UEFI/ACPI is how to find the XSDT.

Point being, there are different methods depending on the firmware on x86 too.

Yup, and then they added a requirement that the APs return to the firmware before calling ExitBootServices(), thereby rendering the whole thing useless. It's almost a good system. I have come to realize that "almost" is just a polite way of saying "not".

Yeah, so true.

Looking at the UEFI membership list, I wonder if this is yet another case of diffusion of responsibility. Can you tell me who broke the spec? No, this way it could have been anyone. And you have no-one to blame.

Correction, you can blame all the UEFI participants equialy

So the best we can do is blindly use some low mem address for the trampoline code and hoping it won't crash the system... What a real progress, well done UEFI!

Cheers,
bzt

ps: BIOS was simpler, (much-much) faster, more stable and safer, cleaner and an OS could use all BIOS services at run-time! It was better in all aspects save it was real-mode. But you could use VM86 mode or libx86emu, so there was a way, while with UEFI you're just screwed. No-one in their sane minds can say UEFI is better! But this is the sh*t forced on all OS developers these days... We have a saying: running around in the d*ckforest with an open mouth, that's what UEFI is!

[nexos]

So the best we can do is blindly use some low mem address for the trampoline code and hoping it won't crash the system... What a real progress, well done UEFI!

You could look at the EFI memory map and find a region under 1M that is free. If none is found, then we have a problem.

[zaval]

But they could be! For example, you could switch the screen resolution from your OS any time using BIOS/VBE. But you can't do that with GOP (being a boot time service). There's absolutely no technical nor hardware related reasons why those protocols are inaccessible for the OS, except shitty UEFI design and implementations.

Then you would be unhappy with the way, that "better" UEFI managed a thing for you and complain, that your OS would do it better. The only thing is needed for such an attitude - is your wish to "laugh" at "intel engineers", that, of course, are "dumb". it's not serious.
The point is, the "technical" reason behind the unavailability of all those things for an OS is that, those things ARE FW itself and for the FW and OSL to boot your OS and nothing more and then your OS gets FULL control over the machine, so you can do whatever you want. they just physically go away from memory, freeing space for your OS to manage the machine. what the OS is supposed to do, in fact. If some protocols were made RTS, then all the protocol management had to be RTS too and stay in memory for all runtime. that would mean more memory would be allocated as RTS and it would need to be preserved during the OS runtime. AND protocol managing is a big part of UEFI, so it would require a lot of memory. for what? for freeing the OS from a routine, naturally belonging to it, of starting up secondary processors? maybe then the FW should take care of other OS burdens? GUI? webrowsers?

So the best we can do is blindly use some low mem address for the trampoline code and hoping it won't crash the system... What a real progress, well done UEFI!

Why "blindly"? You can pick the needed place once you took over the system memory - meaning after ExitBootServices(). Definitely, there will be free space below 1MB. Why you ignore this obvious scenario? UEFI IS a single CPU thing, just as any sane FW should be. bringing up secondary processors IS the OS responsibility!

[bzt]

You could look at the EFI memory map and find a region under 1M that is free. If none is found, then we have a problem.

Yeah, but even if you find a region dynamically, you'll have a problem. There are instructions that must use absolute addresses, not to mention system descriptors like gdt_value etc. Since you cannot mix real mode, prot mode and long mode relocations in an object file, this sucks. You cannot rely on the compiler generating the relocation records for you, must be done manually. Not impossible, just lot of unnecessary work.

I think I'll go with a fixed address instead (like 0xF000 for example) and allocate that using AllocateAddress. If not successful, then display a warning and skip SMP alltogether. Should work in almost every machines, does not crash with faulty UEFI drivers, and I don't have to worry about searching the memory map or relocating code manually.

Cheers,
bzt

[testjz]

Sigh, SMP could in fact be something that can be setup in the bootloader, if not for the UEFI requirements. This is platform initialization after all, and very platform-dependent on top of that, and needs to be done on every machine that has two or more cores. It's nothing like running the init process or starting a web browser. The EFI_MP_SERVICES_PROTOCOL could had been quite useful IMO, but that little detail rendered it useless and I dare say it practically functions as pure bloat at this point, because the UEFI implementers won't bother to test something that's not required to boot Windows or Linux (if they even test the latter), and I doubt any driver uses that.

bzt, I looked a bit through the BOOTBOOT design and implementation yesterday, before I discovered this topic, and also a bit now. It looks interesting to me as an idea, but currently I'd still say that hobby OS developers should write their own bootloader or use a widely tested one. On the upside, though, you seem to have a better design than multiboot (you probably just need to iron it out, same for the implementation), and I suppose it's easier for a hobbyist to contact you here to ask questions about BOOTBOOT (and help you back if they discover an issue, as I'd assume is the case here), than to contact the GRUB developers.

Anyway, in bootboot.c at line 1178, the "0x8000" in CopyMem() should probably be "ap_code", because you might later change the latter and (unlikely but possibly) forget to modify the former, especially if you decide to use the memory map to find a lower address. And at page 20 of the specification, you misleadingly (see note) write under limitations:

As it boots in protected mode, it only maps the first 4G of RAM

which probably should instead be:

If it boots in protected mode, it only maps the fist 4G of RAM

Note: I hope it's misleadingly written and not the reality, though the code seems to confirm my hopes. On the other side again, in the Hungarian edition you say "mivel" which, according to Wiktionary, means "as" in the sense of "because" and not "if" (I don't know Hungarian).

[bzt]

Sigh, SMP could in fact be something that can be setup in the bootloader, if not for the UEFI requirements. This is platform initialization after all, and very platform-dependent on top of that, and needs to be done on every machine that has two or more cores. It's nothing like running the init process or starting a web browser. The EFI_MP_SERVICES_PROTOCOL could had been quite useful IMO, but that little detail rendered it useless and I dare say it practically functions as pure bloat at this point, because the UEFI implementers won't bother to test something that's not required to boot Windows or Linux (if they even test the latter), and I doubt any driver uses that.

I agree!

bzt, I looked a bit through the BOOTBOOT design and implementation yesterday, before I discovered this topic, and also a bit now. It looks interesting to me as an idea, but currently I'd still say that hobby OS developers should write their own bootloader or use a widely tested one.

Agreed. The problem is, Multiboot is a 32 bit protocol (with a 64 bit hack), and Grub is not available on Raspberry Pi at all. I wanted a clean, 64-bit only, multiplatform-by-design loader.

On the upside, though, you seem to have a better design than multiboot (you probably just need to iron it out, same for the implementation), and I suppose it's easier for a hobbyist to contact you here to ask questions about BOOTBOOT (and help you back if they discover an issue, as I'd assume is the case here), than to contact the GRUB developers.

Thanks! The thruth is, many have tested BOOTBOOT, and they open issues on the gitlab's issue tracker. I prefer that over contacting me here, but that's fine too. It's already been used for years now, it's ironed out pretty much

Anyway, in bootboot.c at line 1178, the "0x8000" in CopyMem() should probably be "ap_code", because you might later change the latter and (unlikely but possibly) forget to modify the former, especially if you decide to use the memory map to find a lower address.

Actually no good. The code that's copied to 0x8000 is strictly compiled at that address (see smp.S). The reason for this is because you can't use "org" directive, and without gcc is not smart enough to generate relative offsets. That's why I had to manually calculate all addresses in the relocated trampoline code. If you change the address, then you'll also have to recalculate all those real-mode and prot-mode addresses too in the Assembly. I might have used a define, but loading at an arbitrary address would require run-time relocations for sure.

And at page 20 of the specification, you misleadingly (see note) write under limitations:

As it boots in protected mode, it only maps the first 4G of RAM

which probably should instead be:

If it boots in protected mode, it only maps the fist 4G of RAM

Note: I hope it's misleadingly written and not the reality, though the code seems to confirm my hopes. On the other side again, in the Hungarian edition you say "mivel" which, according to Wiktionary, means "as" in the sense of "because" and not "if" (I don't know Hungarian).

Well, it does all boot process in protected mode. There's only a very-very minimal 64 bit code in the loader (sets up segment registers, stack and then jumps to the kernel's long mode entry point). The same stands for the coreboot implementation, because it also runs in prot-mode. UEFI version on the other hand runs entirely in long mode.

Now maybe the phrasing is not the best. Running the loader in protected mode doesn't need maps over 4G, that's why it maps only 4G. Just for the records, the protocol only guarantees that the initrd will be entirely loaded into the identity mapped area, so 4G is enough, since you simply can't load it above 4G in prot-mode. The reason for this identity mapping is, that all kernels tend to set up their own paging tables anyway, so there's no point in putting lot of effort into boot time mappings.

Cheers,
bzt