[solved] clang not aligning my elf program headers

[kzinti]

So I have some linker script that I use to link my kernel. I specify 3 program headers and align them on page boundaries (4K). This works well with GCC where I link using gcc:

Code: Select all

21:15:34-kzinti@droneship:~/dev/rainbow-os (master)$ readelf -l build/arch/x86_64/kernel/kernel

Elf file type is EXEC (Executable file)
Entry point 0xffffffff8000b708
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000001000 0xffffffff80000000 0xffffffff80000000
                 0x000000000000f35c 0x000000000000f35c  R E    0x1000
  LOAD           0x0000000000011000 0xffffffff80010000 0xffffffff80010000
                 0x00000000000053fc 0x00000000000053fc  R      0x1000
  LOAD           0x0000000000017000 0xffffffff80016000 0xffffffff80016000
                 0x0000000000002013 0x000000000002d002  RW     0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .text 
   01     .rodata .eh_frame 
   02     .data .init_array .fini_array .vdso .bss 

Now I have decided to dump gcc and adopt clang. Here is what I get:

Code: Select all

21:17:53-kzinti@droneship:~/dev/rainbow-os-2/build (master)$ readelf -l kernel/src/kernel 

Elf file type is EXEC (Executable file)
Entry point 0xffffffff80000000
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000001000 0xffffffff80000000 0xffffffff80000000
                 0x000000000000000e 0x000000000000000e  R E    0x1000
  LOAD           0x000000000000100e 0xffffffff8000000e 0xffffffff8000000e
                 0x000000000000007e 0x000000000000007e  R      0x1000
  LOAD           0x0000000000001090 0xffffffff80000090 0xffffffff80000090
                 0x00000000000000e0 0x00000000000000e0  RW     0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .text 
   01     .interp .note.gnu.build-id .dynsym .dynstr .gnu.hash 
   02     .dynamic 

Notice that:
1) The alignment of each program header is "0x1000" as expected.
2) The offset (and virtual address) of the last two program headers is (are) NOT aligned to 0x1000. This I did not expect.
3) Not the same code is compiled in both cases, but that doesn't explain #2

Perhaps my link script needs fixing but I am not sure what. I have been unsuccessful at the googles.

Here is my link script (same used in both cases):
https://github.com/kiznit/rainbow-os/bl ... kernel.lds

[Korona]

Both linkers align correctly: the alignment requirements of the ELF standard demand that p_vaddr % p_align == p_offset % p_align, but not that this expression equals zero. p_align is basically the page size (which can be modified by some command line arguments to ld/lld). This requirements is designed such that it is possible to map the segments from the file into memory (because addresses and offsets match modulo the page size), while still keeping the file size minimal (there is no need to insert padding into the file).

If you want the segments to be 4k-aligned, you have to insert padding manually in your linker script (but that will add a bit of overhead in terms of disk usage).

If you want the segments to start on different pages, it's enough to ensure that the vaddrs are at least 0x1000 apart (e.g., by doing . += 0x1000 in the linker script). This should be enough to properly map the segments at runtime and it's preferable to manual padding.

[kzinti]

If you want the segments to start on different pages, it's enough to ensure that the vaddrs are at least 0x1000 apart (e.g., by doing . += 0x1000 in the linker script). This should be enough to properly map the segments at runtime and it's preferable to manual padding.

Can you clarify how this works? I've tried adding ". += 0x1000" in a few places and it doesn't appear to do anything (some results as before). Also adding 0x1000 to the vaddr doesn't appear to be what i need, I want the vaddr to be aligned to the next page boundary, not moved 0x1000 bytes ahead. Seems to me that the issue is that gcc will link program headers on page boundaries by default and clang doesn't.

Ultimately I want to be different program headers to be aligned to pages so that i can map them with different memory protections. I also need to be able to map the VDSO page(s) in user space: this also needs to be page-aligned and padded to not leak kernel data/code into user space.

Code: Select all

SECTIONS
{
    . = 0xFFFFFFFF80000000;

    .text ALIGN(4K) :
    {
        *(.text*)
    } :phdr_text

    . += 0x1000;

    .rodata ALIGN(4K) :
    {
        *(.rodata*)
    } :phdr_rodata

    . += 0x1000;

    .data ALIGN(4K) :
    {
        *(.data*)
    } :phdr_data

The following seems to almost work:

Code: Select all

SECTIONS
{
    . = 0xFFFFFFFF80000000;

    .text ALIGN(4K) :
    {
        *(.text*)
    } :phdr_text

    .rodata ALIGN(4K) :
    {
        . = (. & ~0xFFF) + 0x1000;
        *(.rodata*)
    } :phdr_rodata

    .data ALIGN(4K) :
    {
        . = (. & ~0xFFF) + 0x1000;
        *(.data*)
    } :phdr_data

Code: Select all

08:11:36-kzinti@droneship:~/dev/rainbow-os-2/build (master)$ readelf -l kernel/src/kernel 

Elf file type is EXEC (Executable file)
Entry point 0xffffffff80000000
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000001000 0xffffffff80000000 0xffffffff80000000
                 0x000000000000000e 0x000000000000000e  R E    0x1000
  LOAD           0x0000000000002000 0xffffffff80001000 0xffffffff80001000
                 0x000000000000107c 0x000000000000107c  R      0x1000
  LOAD           0x0000000000004000 0xffffffff80003000 0xffffffff80003000
                 0x00000000000010e0 0x00000000000010e0  RW     0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .text 
   01     .rodata .interp .note.gnu.build-id .dynsym .dynstr .gnu.hash 
   02     .data .dynamic 

[kzinti]

Accoding to all the info I can find, putting ALIGN() on a section should do what i want it to do (but doesn't work with clang).

See "ALIGN" in the manual here: https://ftp.gnu.org/old-gnu/Manuals/ld- ... /ld_3.html

As an example, to align the output .data section to the next 0x2000 byte boundary after the preceding section and to set a variable within the section to the next 0x8000 boundary after the input sections:

SECTIONS{ ...
.data ALIGN(0x2000): {
*(.data)
variable = ALIGN(0x8000);
}
... }

[linuxyne]

In your original post where the output from clang compilation is shown, there aren't any rodata or data sections created, but these are the sections in the .lds that govern the alignment. Is the absence of those sections which define the alignment a possible cause for the unaligned segments?

[kzinti]

In your original post where the output from clang compilation is shown, there aren't any rodata or data sections created, but these are the sections in the .lds that govern the alignment. Is the absence of those sections which define the alignment a possible cause for the unaligned segments?

Interestingly enough, my current kernel doesn't have any data / rodata in it (!!!). So maybe it's working properly. Let me add some data and see what happens.

[kzinti]

I added some data and rodata and it works just fine. Sounds like it was a 12 inch problem.

Code: Select all

Elf file type is EXEC (Executable file)
Entry point 0xffffffff80000000
There are 3 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000001000 0xffffffff80000000 0xffffffff80000000
                 0x0000000000000017 0x0000000000000017  R E    0x1000
  LOAD           0x0000000000002000 0xffffffff80001000 0xffffffff80001000
                 0x000000000000008c 0x000000000000008c  R      0x1000
  LOAD           0x0000000000003000 0xffffffff80002000 0xffffffff80002000
                 0x0000000000000108 0x0000000000000108  RW     0x1000

 Section to Segment mapping:
  Segment Sections...
   00     .text 
   01     .rodata .interp .note.gnu.build-id .dynsym .dynstr .gnu.hash 
   02     .data .dynamic