Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
This ancient post is the first and the only thing that Google pulls out on this topic. I'm not really surprised - utilizing multiple CPUs is an extremely rare feature in hobby OSes.
Anyway, that thread left two of my questions unanswered.
1. With UEFI, the bootloader starts off in protected mode with A20 enabled and 4 KB identity paging, as well as a system table pointer bundle that includes ACPI which can be used to discover other cores/CPUs. When I wake up a core myself, does the firmware boot it in protected mode with A20 open, or do I have to do GRUB's work myself for the rest of the cores?
2. How do I place the entry point and the GPT in the first 1 MiB of memory without trespassing into UEFI land?
Any suggestions or example code of how to find and wake up other cores from EFI_Main()?
You're not finding any results because UEFI doesn't really change how you initialize multiple CPUs. You'll want to read the Multiprocessor Specification version 1.4 if you haven't already.
To expand upon Octo's answer: If you have access to UEFI, then you have access to the System Table. And with that, access to ACPI. And with that, access to the MADT, which tells you the APIC IDs of all enabled CPUs. And how to proceed from here is something you can read in the Intel SDM, volume 3A, chapter 8.4. Or else in the AMD APM volume 2 somewhere. In all brevity, you are expected to send an INIT IPI to the target processor, wait for 10 ms, send a SIPI to it, wait for 200 µs, check to see if the CPU started, if so, assume success, else send a second SIPI and wait up to 100 ms. If the processor still is not up, it's not coming. (I would send it another INIT IPI on failure, just in case the processor is somehow up and running something I didn't ask it to. INIT IPI should stop it from executing any instructions)
Initial state of the AP will be 16-bit mode, with most registers initialized to 0, including IP. CS will be xx00, where xx is the vector of the SIPI. From there you execute the usual trampoline to end up in your OS.
Since there is only one A20 gate in the system, it will be necessarily enabled already. However, MSRs are processor specific, and thus must be reloaded with their values as needed. This includes MTRRs, if you reprogram those. I have no idea what you want with the GPT in the first MB of memory. I have a normal and a debug trampoline, and even the debug trampoline is less than 3000 bytes. Since the initialization code takes at least one page, anyway, this is all you'll need.
I don't think you can use that service as a substitute for the ACPI MADT, it doesn't give a whole lot of information about the APs.
You certainly can't use it to boot the APs:
UEFI Platform Initialization Specification wrote:The Protocol is available only during boot time.
...
Modules that use this protocol must guarantee that all non-blocking mode requests on all APs have been completed before the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES is signaled.
Octocontrabass wrote:I don't think you can use that service as a substitute for the ACPI MADT, it doesn't give a whole lot of information about the APs.
You certainly can't use it to boot the APs:
UEFI Platform Initialization Specification wrote:The Protocol is available only during boot time.
...
Modules that use this protocol must guarantee that all non-blocking mode requests on all APs have been completed before the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES is signaled.
I think you still can use it at boot time in order to bring the APs to a known state (protected mode with your code running on the AP and waiting for an OS-specific IPI).
If I understand correctly, I would expect the boot-loader to use MpServices in order to execute a 'dispatching user-provided function' as explained in the header file. The user function could be a simple 32-bit or 64-bit piece of code that sends EFI_EVENT to the BSP then waits for an IPI that is specific to the OS.
The BSP (running the UEFI boot-loader) waits for the event to make sure that the AP is initialized. the bootloder then exits boot services and boots into the OS, which communicates the IPIs to the APs, executes the scheduler on them, then the APs have fun together.
According to the spec, the firmware is allowed (and perhaps expected) to reset the APs when you call ExitBootServices().
UEFI Platform Initialization Specification wrote:After the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES is signaled, all the APs must be placed in the OS compatible CPU state as defined by the UEFI Specification. Implementations of this protocol may use the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES to force APs into the OS compatible state as defined by the UEFI Specification.
The UEFI specification says the OS compatible state for x86 is uniprocessor, which means firmware will place all APs in the wait-for-SIPI state.
Octocontrabass wrote:According to the spec, the firmware is allowed (and perhaps expected) to reset the APs when you call ExitBootServices().
UEFI Platform Initialization Specification wrote:After the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES is signaled, all the APs must be placed in the OS compatible CPU state as defined by the UEFI Specification. Implementations of this protocol may use the UEFI event EFI_EVENT_LEGACY_BOOT_GUID or EFI_EVENT_GROUP_EXIT_BOOT_SERVICES to force APs into the OS compatible state as defined by the UEFI Specification.
The UEFI specification says the OS compatible state for x86 is uniprocessor, which means firmware will place all APs in the wait-for-SIPI state.
Thanks Octocontrabass! In that case I would rather prefer to implement a trampoline that brings the AP from real-mode to protected/long mode (just like what all people do), and never touch the APs before Exit Boot Services. I think this is the safest way.
Octocontrabass wrote:You're not finding any results because UEFI doesn't really change how you initialize multiple CPUs. You'll want to read the Multiprocessor Specification version 1.4 if you haven't already.
Is there really ANY reason at all why you would bother with the Multiprocessor Specification (unless you run on really old hardware, but that is typically single-core anyway)?
My approach is that it is ACPI that provides the relevant information for how many cores & processors a machine has, and how they connect to each other and to PCI devices.
rdos wrote:Is there really ANY reason at all why you would bother with the Multiprocessor Specification (unless you run on really old hardware, but that is typically single-core anyway)?
Last I checked, the ACPI specification doesn't explain how to bring APs online.
rdos wrote:Is there really ANY reason at all why you would bother with the Multiprocessor Specification (unless you run on really old hardware, but that is typically single-core anyway)?
Last I checked, the ACPI specification doesn't explain how to bring APs online.
That's only for informational purposes. I doesn't add anything to how to decide about how a particular system is configured.
rdos wrote:Is there really ANY reason at all why you would bother with the Multiprocessor Specification (unless you run on really old hardware, but that is typically single-core anyway)?
Perhaps, if you don't know both the Intel SDM and the AMD APM tell you how to initialize an AP if you know its local APIC ID? That used to be standardized in the MP spec only. And to this day it contains information considered relevant by historians, such as how to initialize secondary CPUs if those are 486s, which was that stuff with the INIT level de-assert, which I didn't fully absorb, either, since it is about as relevant today as a punch card.
The AP initialization sequence is basically covered in the APIC description (which is included in the Intel & AMD manuals). Maybe specific delays are not there, but then I'm not sure using exactly those is really necessary. Of course, that assumes the processor use the APIC, and not the legacy PIC, which has no support for AP initialization. AFAIK, 486 processors always came with PIC, and so would need some other initialization process. The inter-processor communication also is different since modern processors use the APIC for this too, and the legacy PIC doesn't support this.
So, this code should be part of the APIC driver in an OS. If some other interrupt controller comes along, then a new method can be implemented in that controller driver instead. The 486 logic will need some specific driver that knows how that worked.
nullplan wrote:Perhaps, if you don't know both the Intel SDM and the AMD APM tell you how to initialize an AP if you know its local APIC ID?
The Intel SDM is very large, so it's possible I missed it, but I was under the impression that it only explained how firmware should bring APs online, and didn't cover how the OS should do so after firmware has initialized everything.
nullplan wrote:And to this day it contains information considered relevant by historians, such as how to initialize secondary CPUs if those are 486s, which was that stuff with the INIT level de-assert, which I didn't fully absorb, either, since it is about as relevant today as a punch card.
Before the APIC was integrated into the CPU, those IPIs were used to turn the reset signal on and off. You had to leave the signal on for some minimum amount of time for the CPU to reset properly, and the APIC didn't know how long that should be.
rdos wrote:AFAIK, 486 processors always came with PIC, and so would need some other initialization process.
The 486 didn't come with any interrupt controller. It was up to the motherboard to provide APICs if it wanted to support two or more 486 processors.
486 systems with APICs simply would not run most operating systems or applications of the time. This was when PIT & PIC were standard and must-haves in every PC, and programs used them extensively too, not just BIOS and the OS. When the APIC was introduced, it had a PIC emulation mode that was default after boot. More capable OSes that wanted to use multiple cores had to abondon the PIC and start using the APIC. However, when 486 chips were sold, the multiprocessor market was a small niche, and so mainstream systems contiuned to use the PIC. I still use the PIC when booting with an old single core processor regardless if it has an APIC or not. I also ignore ACPI on those systems.
There is a peculiarity about booting an AP core on a PC. Since it is the BIOS that gets control when you boot the AP, you need to tell the BIOS to jump to another location rather than the normal boot location. You do this by setting a byte in the CMOS RAM.
