OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Mar 28, 2024 1:00 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 13 posts ] 
Author Message
 Post subject: Truly portable executables (bootable PE32)
PostPosted: Fri Apr 30, 2010 2:10 pm 
Offline
Member
Member

Joined: Thu Jan 14, 2010 5:35 pm
Posts: 31
Original contest was at http://forum.osdev.org/viewtopic.php?f=2&t=21791. No entries :(

I am attaching my own pe2boot utility with a couple of examples for MSVC, MinGW/GCC, and TinyCC.

Resulting PEs run on Windows and Linux and are bootable on a bare system from CDROM, USB, or PXE/BOOTP.

Runtime stub runs on a bare hardware or under DOS and handles IRQs, Exceptions, and int86() - and it all takes <1.3k! :)


Attachments:
pe2boot_20110704.zip [50.49 KiB]
Downloaded 144 times


Last edited by technik3k on Mon Jul 04, 2011 12:10 am, edited 4 times in total.
Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sat Jul 02, 2011 3:13 pm 
Offline
Member
Member

Joined: Thu Jan 14, 2010 5:35 pm
Posts: 31
The updated version of pe2boot is here and has new features:

    1. Ability to boot directly from network (via PXE/BOOTP)
    2. Can create bootable CDROM images in both "No Emulation" and "1.44 Emulation" modes.
    3. Can make bootable USB sticks (then just copy your executable to the root directory and it boots!)
    4. Better executable compression and updated C and ASM examples for MinGW/gcc, fasm, tcc, and msvc compilers.

Most of barebone examples rely on GRUB. Unlike them pe2boot makes a single executable that can boot from a variety of sources, such as network (PXE/BOOTP), USB drives, CDROMs, Floppies, or run natively inside Windows and Linux operating systems. Another distinct feature is that you don't have to worry about compiler specific static linking rules. Rebasing is done dynamically during decompression.

This is hello.c example with gcc and msvc entry point declarations (other compilers like tcc or bcc32 could be used with small tweaks as well):
Code:
#include <stdio.h>
#include <stdlib.h>

#ifdef __MINGW32__
void debug(void){asm("int $1");}   void __stdcall DllMainCRTStartup(int a, int b, int c)
#else //__MSVC__
void debug(void){__asm {int 1};}   void mainCRTStartup(int a)
#endif
{
   char *p = "Hello World";
   char *v = (char*)(0xB8000+0x6c7);

   if( (int)&a < (int)v )      // if stack < 640k - it is not Win32
   {
      for(;v++,*v++=*p++;);   // write directly to the video memory
      debug();                // invoke debug monitor built into the stub
   }
   else // Win32
   {
      printf(p);
      exit(0);
   }
}

You may use virtually any compiler that can generate PE with base relocations:
Code:
   C:\test>cl hello.c /link /fixed:no /subsystem:console /nodefaultlib msvcrt.lib
   C:\test>c:\mingw\bin\gcc -s -shared -nostdlib hello.c -o hello.exe -lmsvcrt
   tux$ i586-mingw32msvc-gcc -s -shared -nostdlib hello.c -o hello.exe -lmsvcrt

Now you need to compress hello.exe and create bootable USB or CDROM images (pe2boot.exe will run either on Windows or Linux with Wine):
Code:
   pe2boot.exe -s pe_stub.bin  -i hello.exe -o hello.exe     <- compress PE

   pe2boot.exe -b boot_fat.bin -i hello.exe -CD0 hello.iso   <- No emulation
   pe2boot.exe -b boot_fat.bin -i hello.exe -CD2 hello.iso   <- 1.44 emulation
   pe2boot.exe -b boot_fat.bin -i hello.exe -FLP hello.flp   <- 1.44 floppy
   pe2boot.exe -b boot_fat.bin -L "HELLO   EXE" -FAT \\.\U:  <- USB drive
   copy hello.exe u:\    <- if you boot from USB drive it will load hello.exe

Finally, you are ready to test how it runs or boots on various environments:
Code:
  hello.exe                                 <- "Hello World" Win32/Linux
  qemu -fda hello.flp                       <- "Hello World" boot floppy
  qemu -cdrom hello.iso                     <- "Hello World" boot cdrom
  qemu -tftp . -bootp hello.exe -boot n     <- Direct boot with BOOTP

Plans for the near future are to be able to create hybrid USB/CDROM images like ISOLINUX.

Please, let me know if you have any specific comments, suggestions, or feature requests. If you are about to comment, that linker scripts aren't hard and static linking rocks,etc... Please, remember that the goal of this project is truly portable executable that can run on the bare hardware as well as inside an existing OS. In this scenario static linking is not a good option.


Last edited by technik3k on Sat Jul 02, 2011 8:44 pm, edited 2 times in total.

Top
 Profile  
 
 Post subject: Re: Contest: easy relocateable bootable 32-bit PEs & ELFs
PostPosted: Sat Jul 02, 2011 4:03 pm 
Offline
Member
Member
User avatar

Joined: Wed Dec 01, 2010 3:41 am
Posts: 1761
Location: Hong Kong
Sound like your're working on an binary compatibility layer (For example, Linux Binary Compatibility).
I'm surprised you do it on bare-bone, since most people would implement it last when the OS has richer functionality.

Anyway, I'm interested on how many functions you provide in the bulit-in runtime, since it affect the usability on the whole environment.


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sat Jul 02, 2011 8:22 pm 
Offline
Member
Member

Joined: Thu Jan 14, 2010 5:35 pm
Posts: 31
bluemoon wrote:
Sound like your're working on an binary compatibility layer (For example, Linux Binary Compatibility).
I'm surprised you do it on bare-bone, since most people would implement it last when the OS has richer functionality.

Anyway, I'm interested on how many functions you provide in the bulit-in runtime, since it affect the usability on the whole environment.

Actually, I am not doing that myself, but use this concept on Linux.

Pe2boot's output is a PE32 executable. The runtime it adds to an EXE is only 1216 bytes. Think of it as an empty shell for your app with several handles bolted outside that different loading systems can hook to.

When run on Windows it simply lets you call Win32 functions that you normally import from various DLLs.

When Linux loader sees PE32 signature it calls Wine loader, which handles any Win32 calls. In the hello.c example those are printf() and exit() normally found in MSVCRT.dll just as it would happen on Windows. In addition to Win32 functions, you can also call Linux syscalls via "int 80h" (unless it is blocked by security policy).

When you run on DOS or bare hardware, the runtime will detect available memory, and switch CPU into protected mode and call your app. Your interface to the outside world is int86(regs,regs) function and any port I/O, if you like so.

Normally, if you want to make a system utility that runs and boots on various platforms, you would need to create separate:

    Win32 executable to run on Windows 9x, XP, Vista, 7 (or their server siblings)
    ELF executable to run on Linux, FreeBSD, etc...
    DOS executable with some sort of 32-bit extender for FreeDOS, MS-DOS, etc...
    Bare hardware version of the executable packaged onto
      Bootable CDROM image
      Bootable Floppy/USB image
      Image for network boot using PXE/BOOTP
      Boot from HD partition, CHS/LBA, MBR/EFI, large sector size headaches...

Things will get complicated fast...

If you want to limit number of binaries the first logical step is to use PE32 because, thanks to Wine developers, virtually any linux box could run it or is one apt-get away from being able to.

Then to take care of DOS and direct bare hardware boot you need a small runtime that switches CPU into protected mode. This has been done by many people for the last 25 years (80386 was introduced by Intel in 1985). What I did was adding an integrated dynamic loader that could rebase Win32 code to any address and made entire thing ridiculously small (1216 bytes).

Finally, for various boot methods there are few tricks inside runtime itself and and additional 512 byte FAT1x boot sector that works on CDROMs, Floppies, USB sticks, and HardDrives of various sizes and geometries. It is possible to create a hybrid image that can be both burned onto CD/DVD or rawwritten onto floppy or USB. When you boot from it your app.exe boots, when you insert it under an OS you can run app.exe under that OS.

Conceptually, it might even be possible to create a binary that is a valid PE32 executable, valid PXE/BOOTP binary, valid OPTION ROM image AND a valid hybrid bootable CDROM/Floppy/USB/HD image all at the same time. But that would be an overkill :D Any takers?


Top
 Profile  
 
 Post subject: Re: Contest: easy relocateable bootable 32-bit PEs & ELFs
PostPosted: Sat Jul 02, 2011 8:40 pm 
Offline
Member
Member

Joined: Thu Jan 29, 2009 9:13 am
Posts: 95
Very cool stuff. Barring Windows support, could the same thing be made with ELF files?


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sat Jul 02, 2011 9:03 pm 
Offline
Member
Member

Joined: Thu Jan 14, 2010 5:35 pm
Posts: 31
Merlin wrote:
Very cool stuff. Barring Windows support, could the same thing be made with ELF files?


Yes, probably, it is possible. I don't know know enough about ELF file format and dynamic loading to say for sure. Also, just like MultiBoot, PXE/BOOTP has specific requirements of where the magic number and header should reside. I was able to squeeze them into MZ/PE32 header, but it might be a problem with ELF.

As I said in the previous post, my goal is to run everywhere and running PE32s on Linux is only one apt-get away, while running ELFs on Windows is a major headache. It is much harder to install and configure Cygwin or similar emulator than Wine and you will be limited to 64k with .com files under DOS.


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sat Jul 02, 2011 9:41 pm 
Offline
Member
Member
User avatar

Joined: Wed Dec 01, 2010 3:41 am
Posts: 1761
Location: Hong Kong
technik3k wrote:
When run on Windows it simply lets you call Win32 functions that you normally import from various DLLs.

When Linux loader sees PE32 signature it calls Wine loader, which handles any Win32 calls. In the hello.c example those are printf() and exit() normally found in MSVCRT.dll just as it would happen on Windows. In addition to Win32 functions, you can also call Linux syscalls via "int 80h" (unless it is blocked by security policy).

When you run on DOS or bare hardware, the runtime will detect available memory, and switch CPU into protected mode and call your app. Your interface to the outside world is int86(regs,regs) function and any port I/O, if you like so.


I'm a bit confused, although the resulting binary is a single file, you still have to write different code segments for different target platform and branch internally? What will happen if I called winapi in the code and the program loaded on bare-bone? do it fail to resolve symbols or just call a stub function and result fail? How about I use in86() and run the program under windows?

What's the advantage over the following environment detection stub + zip/concat solution?
binary layout:
stub
[executable map]
PE executable for Windows, with MZ stub for DOS
executable for Linux

workflow:
1. The loader pass control to stub
2. stub detect environment
3. get offset of suitable executable from the map
4. perform initialization, relocation and pass control to the suitable executable


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sat Jul 02, 2011 10:20 pm 
Offline
Member
Member

Joined: Thu Jan 14, 2010 5:35 pm
Posts: 31
bluemoon wrote:
I'm a bit confused, although the resulting binary is a single file, you still have to write different code segments for different target platform and branch internally? What will happen if I called winapi in the code and the program loaded on bare-bone? do it fail to resolve symbols or just call a stub function and result fail? How about I use in86() and run the program under windows?

What's the advantage over the following environment detection stub + zip/concat solution?
binary layout:
stub
[executable map]
PE executable for Windows, with MZ stub for DOS
executable for Linux

You are right about workflow, and that is what I am doing. The difference from your layout is that there is no executable for Linux. Thanks to Wine, I could run native Win32 executable on Linux. The switch happens when I am under DOS or BIOS. Here is my layout:

    MZ header
    -> BOOTP entry point
    ...
    PE32 Header
    -> DOS entry point
    ...
    BOOTP magic
    -> BIOS CDROM/USB/HD boot entry point
    ...
    commonBOOTP_DOS_BIOS:
      Detect memory, A20, XMS
      Initialize GDT, IDT, IRQ handles
      Switch CPU to 32-bit protected mode
    jmp common32
    -> Win32 entry point
    common32:
      Decompress and rebase executable
      if Win32 foreach DLL { LoadLibrary(); GetProcAddresses(); }
      jmp app entry point
    (compressed payload app)

Then one of arguments passed to the app is environment id. If you call Win32 function under DOS you crash. If you call int86() under Win32 you crash. Implementing a layer that provides common functionality depending on the platform is the subject of the next project. What I did so far is just built a container that is recognized as native on the various platforms.


Last edited by technik3k on Sat Jul 02, 2011 10:31 pm, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sat Jul 02, 2011 10:30 pm 
Offline
Member
Member
User avatar

Joined: Wed Dec 01, 2010 3:41 am
Posts: 1761
Location: Hong Kong
I see, thanks for the explanation, that make senses now.


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sun Jul 03, 2011 12:35 am 
Offline

Joined: Sat Mar 27, 2010 2:15 am
Posts: 24
Looks very cool!
Just wondering..... what license is the code released under? There seems to be no license info in the download, and none in the source files that I looked at.
- m


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sun Jul 03, 2011 3:30 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9301
Location: On the balcony, where I can actually keep 1½m distance
Quote:
When Linux loader sees PE32 signature it calls Wine loader, which handles any Win32 calls. In the hello.c example those are printf() and exit() normally found in MSVCRT.dll just as it would happen on Windows. In addition to Win32 functions, you can also call Linux syscalls via "int 80h" (unless it is blocked by security policy).

When you run on DOS or bare hardware, the runtime will detect available memory, and switch CPU into protected mode and call your app. Your interface to the outside world is int86(regs,regs) function and any port I/O, if you like so.
So you still need to write two applications. Plus that any attempt to use printf or anything like it fails because it will either give a linking conflict if you provide your own (something you need to do for bare metal support) or you use windows' and you will most likely crash on bare metal.

In other words, welcome to Jurassic Park. Why aren't the raptors in their enclosure?

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sun Jul 03, 2011 11:02 am 
Offline
Member
Member

Joined: Thu Jan 14, 2010 5:35 pm
Posts: 31
mooseman wrote:
Looks very cool!
Just wondering..... what license is the code released under? There seems to be no license info in the download, and none in the source files that I looked at.
- m

It is released under ISC license. Two main sources files have it. I will add a separate licence file as well.

Combuster wrote:
So you still need to write two applications. Plus that any attempt to use printf or anything like it fails because it will either give a linking conflict if you provide your own (something you need to do for bare metal support) or you use windows' and you will most likely crash on bare metal.

In other words, welcome to Jurassic Park. Why aren't the raptors in their enclosure?

So what would be your suggestion? If you intend to have a utility running on multiple platforms you have to deal with multiple backends anyway.


Top
 Profile  
 
 Post subject: Re: Truly portable executables (bootable PE32)
PostPosted: Sun Jul 03, 2011 4:13 pm 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9301
Location: On the balcony, where I can actually keep 1½m distance
technik3k wrote:
So what would be your suggestion? If you intend to have a utility running on multiple platforms you have to deal with multiple backends anyway.
The obvious thing: write a program in strict ANSI C. The only deal with multiple backends is that I need to compile more often, but my code is shorter since it's oblivious to backend implementation details and it will run on more platforms than what your tool can ever hope to provide me.

That doesn't mean it's not a nice gimmick, but it's also just that: a nice gimmick, which like Raptors should not roam in the wild because it will lead to accidents :wink:.



Edit: Do you know Java? :twisted:

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 26 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group