OSDev.org

The Place to Start for Operating System Developers
It is currently Tue May 28, 2024 2:16 am

All times are UTC - 6 hours




Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: What are the hidden complexities of binary compatibility?
PostPosted: Sun Apr 28, 2024 12:30 pm 
Offline
Member
Member
User avatar

Joined: Sun Oct 11, 2020 9:46 pm
Posts: 374
Location: United States
Often, beginners in OSDEV imagine their OS being compatible with all the major operating systems. However... wine is the best (not an emulator!) compatibility layer we have in this field so far, and it is still notorious in 2024 for being unreliable. Sure, it's made substantial progress, but many Windows programs that many users find very important (Adobe Creative Cloud, Visual Studio, ROBLOX) still do not work reliably.

There have been many notable attempts at supporting another operating system's software, to varying degrees of success. The (esoteric) Longene project, the entirety of the 20-year-old ReactOS project, Wine, and even Managarm who aims for source compatibility rather than binary compatibility.

On the surface, it seems relatively simple. Support another system's syscall interface (of which, many do not conflict, as Linux uses interrupt 0x80, and Windows uses interrupt 0x2a IIRC, making multi-compat possible), and the rest of the cards should fall in line, right?

Obviously not, but why?

_________________
Skylight: https://github.com/austanss/skylight

I make stupid mistakes and my vision is terrible. Not a good combination.

NOTE: Never respond to my posts with "it's too hard".


Top
 Profile  
 
 Post subject: Re: What are the hidden complexities of binary compatibility
PostPosted: Sun Apr 28, 2024 1:15 pm 
Offline
Member
Member

Joined: Wed Aug 30, 2017 8:24 am
Posts: 1618
"Just support the syscall layer". You say that as though it was simple.

Supporting compatibility with Linux for example would require you to support all of Linux' system calls to some degree at least. That includes SYS_fork and SYS_clone. With those alone, you have already made the choice that processes/threads are created in the fork style, as copies of their parents, with only select things changed. It also requires you to support POSIX file descriptors, and rdos has previously dumped entire buckets of bile onto that concept, so it is not always well-received.

If you want to also support Windows system calls, you also have to support Windows concepts. Such as drives. I just made a topic on BIOS updates for an HP laptop. The HP program I need to run for this has a function to write its files onto a USB stick. It somehow queries all drives in the system to find a suitable candidate to do the dumping on. Now, this does not work at all on a UNIX-like system, which handles devices completely differently.

Next problem you will have is that the actual Windows system call layer is completely undocumented. It is hidden beneath at least two layers of DLLs, which are allowed to change without notice and frequently do. Of those, only the interface of the highest layer is defined, but a lot of applications actually use the lower layers.

The info you have on the interrupt numbers is outdated; all remotely modern operating systems use syscall or sysenter. So now the system calls may intersect! Although that issue can be taken care of with a "personality" mechanism like Linux has (basically you set a flag on the process, and that flag is preserved across execve() and inherited into children by default),

The next issue you have is that many applications use undocumented "features", or just observed behavior, that you also have to implement in order to not break. For example, on Linux, there is a select() function. That function is defined in POSIX to take a struct timeval as timeout. All other functions take a timeout as a timespec structure, and all of those error if the structure given is invalid. Not select(). It will normalize the timeout argument. Or at least, it did historically. Now POSIX doesn't say that that will happen, but if you implement Linux syscall compatibility, you have to implement stuff like this as well.

And finally, sometimes emulation is just not feasible within your framework. It is no coincidence that many of the applications you mention are graphically intensive. Windows has DirectX, a proprietary graphics library. And on Linux, you have basically no good way to implement it. Emulating DirectX with OpenGL is like emulating a Gamecube with a PS2. They are just two very different approaches.

_________________
Carpe diem!


Top
 Profile  
 
 Post subject: Re: What are the hidden complexities of binary compatibility
PostPosted: Mon May 06, 2024 4:45 pm 
Offline
Member
Member
User avatar

Joined: Mon May 22, 2017 5:56 am
Posts: 826
Location: Hyperspace
austanss wrote:
There have been many notable attempts at supporting another operating system's software, to varying degrees of success. The (esoteric) Longene project, the entirety of the 20-year-old ReactOS project, Wine, and even Managarm who aims for source compatibility rather than binary compatibility.

I think it's worth noting that you've overlooked the whole POSIX ecosystem, of which Managarm here is just the most recent of possibly hundreds of OSs. POSIX and the preceeding Standard Unix standardised the many Unix-like and Unix-compatible OSs which appeared in the 80s if not the late 70s. Compatibility has always been source-level only and has always required patches, but at least you can patch. It has its difficulties but I find it less painful than any other form of compatibility. Well, no, there are frameworks, sometimes attached to languages, which try to work the same way on every OS they support. Those are nice, though there are gotchas such as Windows file semantics and performance characteristics being substantially different from POSIX's.

_________________
Kaph — a modular OS intended to be easy and fun to administer and code for.
"May wisdom, fun, and the greater good shine forth in all your work." — Leo Brodie


Top
 Profile  
 
 Post subject: Re: What are the hidden complexities of binary compatibility
PostPosted: Wed May 08, 2024 6:54 pm 
Offline
Member
Member
User avatar

Joined: Thu Nov 16, 2006 12:01 pm
Posts: 7615
Location: Germany
nullplan wrote:
Windows has DirectX, a proprietary graphics library. And on Linux, you have basically no good way to implement it. Emulating DirectX with OpenGL is like emulating a Gamecube with a PS2. They are just two very different approaches.


Nitpick: Proton, which is effectively a translation from DirectX to Vulkan, works pretty fine.

_________________
Every good solution is obvious once you've found it.


Top
 Profile  
 
 Post subject: Re: What are the hidden complexities of binary compatibility
PostPosted: Mon May 27, 2024 11:22 pm 
Offline
Member
Member

Joined: Fri Nov 17, 2006 5:26 am
Posts: 262
austanss wrote:
Often, beginners in OSDEV imagine their OS being compatible with all the major operating systems.

Are you sure they do that? Windows is millions of lines of code - none of it written for fun. Who are these people who think they can replicate that on their own? I doubt many programmers who actually program would make a claim that they could write and debug millions of lines of code on their own.

Quote:
However... wine is the best (not an emulator!) compatibility layer we have in this field so far, and it is still notorious in 2024 for being unreliable. Sure, it's made substantial progress, but many Windows programs that many users find very important (Adobe Creative Cloud, Visual Studio, ROBLOX) still do not work reliably.

What is your scope? I believe WINE runs anything that conforms to published specs. If you choose to run software that goes beyond the actual documented API - whose fault is that?

Quote:
There have been many notable attempts at supporting another operating system's software, to varying degrees of success. The (esoteric) Longene project, the entirety of the 20-year-old ReactOS project, Wine, and even Managarm who aims for source compatibility rather than binary compatibility.

On the surface, it seems relatively simple.

Programming anything substantial is rarely simple. Are you a programmer who thinks it is simple? How much would you charge me to write a C compiler? It will require about 10,000 lines of code. Far less than the number of lines of code in Windows.

Quote:
Support another system's syscall interface (of which, many do not conflict, as Linux uses interrupt 0x80, and Windows uses interrupt 0x2a IIRC, making multi-compat possible), and the rest of the cards should fall in line, right?

Obviously not, but why?


The published API for Windows is to use DLLs, not an interrupt.

Regardless, what's your scope? PDOS/386 runs both Linux programs that use int 0x80, and Windows programs that use kernel32.dll or msvcrt.dll. It has the OS/2 API (also DLLs too). And another interface called PDOS-generic.

The supported API is a subset of what Windows etc supports. A subset that can be achieved by a single programmer. And sufficient to support an entire toolchain to allow self-hosting.

So yes - if you show me the programmer you refer to who wants compatibility, and show me to what extent he wants that compatibility - then yes, it can be done already. And has been done already. WINE is just one example. It already satisfies any reasonable goal. Way above what is reasonable for a single programmer to achieve.


Top
 Profile  
 
 Post subject: Re: What are the hidden complexities of binary compatibility
PostPosted: Mon May 27, 2024 11:30 pm 
Offline
Member
Member

Joined: Fri Nov 17, 2006 5:26 am
Posts: 262
nullplan wrote:
Supporting compatibility with Linux for example would require you to support all of Linux' system calls to some degree at least. That includes SYS_fork and SYS_clone.

If you allow "to some extent" ...

Quote:
With those alone, you have already made the choice that processes/threads are created in the fork style, as copies of their parents, with only select things changed.

... then this is not strictly true.

PDPCLIB uses:

CLONE_VFORK 0x4000

so the Unix model isn't really required.

BFN. Paul.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Google [Bot], Majestic-12 [Bot] and 10 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group