OSDev.org

I'm writing an OS for ARM. The OS is compiled to run at a very high virtual address. One of the first things I do, using only PIC while running in physical address space, is setting up some page tables which map the OS image and allocate a few more pages for stack and such. To jump to a high address in virtual space, I also create an identity mapping of the OS image so that when I enable the MMU I don't get an exception and am able to jump to a known virtual address. After this, I remove this identity mapping from the page tables.

So, this works, but I'm wondering if there are any "standard" techniques to do this. What do mainstream OSes such as Linux do? What do you do in your implementations?

Thank you in advance

Hi,

Pretty much that's all. There are not much magic here, you must have an identity mapping and a higher half mapping at the same time before you jump. That's not just standard, but it is a must, otherwise you get a prefetch abort.

Here's a really good doc with source: https://github.com/s-matyukevich/raspberry-pi-os/blob/master/docs/lesson06/rpi-os.md
Linux does not do much either (except for KASLR when it applies dynamic relocations): https://github.com/torvalds/linux/blob/master/arch/arm64/kernel/head.S#L848

Cheers,
bzt

Sorry for digging this up, but I have another question. One problem with this approach is when you have the physical address of where the original image was loaded overlapping the virtual address range to where you compiled your code. What is the "standard" solution to this problem?

Hi,

No worries.
This is not an approach, this is the only possible way:
In identity mapping, you have a code that enables the MMU and jumps to the virtual address. Let say this code is on page A.
In the virtual map, you MUST map page A identically, otherwise the CPU will throw a prefetch abort as soon as you enable the MMU. No workarounds possible.
Once you are running code from a virtual address, then you can remove the identity mapping for page A.

This also means overlapping your kernel with the boot code in general is a really bad idea. In the virtual map, you must map the page with your MMU enabling code. In theory you could write a special page B in the kernel, which has only one jump instruction at the same position as page A, and you map it in place of page A when you enable the MMU, but I don't think crafting such a special page worth it.

If you really must use the same area for the loader and the kernel, then you can always relocate a small function to a safe address (let's say 0x1000) out of the way, and map that page identically. That function would enable the MMU and jump to the final virtual address, that's all.

Cheers,
bzt

Very clear explanation! Thank you for your reply!

Why not build the system such that only the bootloader gets identity mapped and it reads the kernel's start address out of the kernel image? It's what I do (on the PC). Also, on PowerPC there is an instruction to switch on the MMU and transfer control to a different address, all in one go. So there, no identity mapping is ever needed. But as far as I know, PowerPC is the only architecture with this feature.

Yeah, that's what I do too. I think the OP's problem is (the reason to bring up overlapping memory) that they want to map the kernel where the bootloader is (so that the kernel virtual address overlaps with the bootloader's physical address). That won't work, but that relocated function workaround can solve it.
Works on RPi too. It is a good, portable solution, which does not assume nothing on the actual MMU hardware.
You're absolutely correct. The only reason why this thing is causing issues is because the MMU enabling instruction and the following jump instruction are two separate instructions. Aka. the CPU needs to read that second jump instruction after the MMU is already enabled, but the instruction pointer is still pointing to a physical address. If they were a single instruction, there would be no problem at all.

Cheers,
bzt

I don't get what you mean. I'm using U-boot as a bootloader. When it handles control to the OS, the OS is running with MMU disabled. I also don't want any dependencies between my kernel and the bootloader and I want it to be possible to load the kernel image anywhere, hence my question.



I was actually wondering about how such an instruction would be useful! I didn't know there was an ISA that actually featured it. The only downside I guess would be that such an instruction would only be useful to do exactly this and only executed one time.

What about position-independent code (PIC)? Also, out of curiosity, is this for arm32 or aarch64?

I believe having the complete kernel compiled with PIC would have too much overhead. This is for aarch64.

Honestly I don't think so. The LDR literal instruction (for aarch64), which uses PC-relative addressing mode, is frequently generated by the compiler, and doesn't induce any overhead:


You can also load literals using:


And the assembler will allocate this literal for you (placed in text section, close to the instruction itself), and will generate LDR literal instruction for you (PC-relative).

In general, PC-relative addressing mode can be implemented in the pipeline without causing any stalls (except for memory/cache read/write delays, which could happen with any addressing mode). The size of the instruction is 32 bits so it doesn't constitute any special instruction-fetch stalls.

This young boy is also useful:


The compiler generates it to load the absolute address of a variable into a register. It is -kind of- similar to LEA of x86 (kind of).

After loading the absolute address into some register, you can use 'LDR register' and 'STR register' instructions. In aarch64, you have to load the address in some register before you use LDR and STR anyway (except for LDR literal), that's why the ISA designer provided you with this cool instruction.

That being said, aarch64 code is inherently position-independent.

My kernel (the main binary) is an ELF file, and as such requires loading like any ELF file (mapping the PT_LOAD segments where required and zeroing out the difference between p_memsz and p_filesz). If the bootloader does not provide this service, the I need to write something that does. So I also have an auxiliary binary I call the adapter, which adapts my kernel's square peg to the main bootloader's round hole. I have a variety of adapters that are capable of starting where quite a few bootloaders leave off and getting my kernel up and running (alright, at the moment I have two, one for multiboot and one for UEFI, but the principle is extensible). The main executable remains the same in the face of such changes.

PowerPC has this quirk, that it disables the MMU on any exception and interrupt. Most operating systems will use the first-level exception handler to save registers and then jump to the main kernel. The instruction I was alluding to is "rfi", return from interrupt. It will copy the low 16 bits of SRR1 into the MSR, and all of SRR0 into the PC (although the ISA doesn't call it that, it calls it the "NIP", next instruction pointer). The machine status register contains a number of important bits, one being privilege level, and two others being "data translate" and "instruction translate". Because the MMU can be enabled separately for data and instruction accesses.

Thus, rfi allows you to enable MMU and change instruction pointer at the same time. It also allows you to do tons of other stuff.