OSDev.org

The Place to Start for Operating System Developers
It is currently Thu Apr 18, 2024 4:22 pm

All times are UTC - 6 hours




Post new topic Reply to topic  [ 60 posts ]  Go to page Previous  1, 2, 3, 4
Author Message
 Post subject: Re: Design Choices of my OS
PostPosted: Fri Mar 06, 2015 11:47 pm 
Offline
Member
Member
User avatar

Joined: Wed Jan 06, 2010 7:07 pm
Posts: 792
So you do want the VM to guess what the virtual drivers are doing? Blitting, rotation, etc. aren't really visible at the hardware level you describe, and the same is true of any feature that your virtual hardware doesn't include but that new physical hardware does (color screens! higher refresh rates! holographic displays!).

Unless what you meant by virtual drivers having UIDs was that the VM would in fact recognize them and include an interface (aka HLE hooks) specifically for them, at which point the hardware interface really isn't set in stone anymore anyway- instead of depending on Nvidia hardware you'll just depend on a particular (version of) HLE hook.

That's virtually identical to the already-existing difference between vendor-specific OpenGL extensions and vendor-agnostic ones, on an implementation that provides software emulation. So your problem would be solved much more simply by just not allowing vendor-specific extensions in your API, and versioning things properly. ;)

_________________
[www.abubalay.com]


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Fri Mar 06, 2015 11:55 pm 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
Rusky wrote:
Unless what you meant by virtual drivers having UIDs was that the VM would in fact recognize them and include an interface (aka HLE hooks) specifically for them, at which point the hardware interface really isn't set in stone anymore anyway- instead of depending on Nvidia hardware you'll just depend on a particular (version of) HLE hook.

Bingo, but like I said, anything inside of the VM is completely blind, deaf, and numb to anything outside of the VM. You can't depend on hooks, they're merely meant to accelerate, not provide functionality (it's impossible for the VM software to even detect if it has been hooked). The point of the virtual hardware is to make this possible. It doesn't matter if there are no hooks for a certain function because it can simply fall back on LLE of the virtual hardware. Thus, all VM software works on all platforms and all physical hardware combinations, guaranteed. The only factor is performance, which is improved via hooks.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 1:25 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4594
Location: Chichester, UK
SoulofDeity wrote:
iansjack wrote:
It seems significant to me that almost every example that you come up with relates to emulating gaming consoles.

That's because that's what I'm talking about.

OK, that makes things a lot clearer. I hadn't realized that this was an OS aimed at games consoles. That makes life a lot easier as you are supporting a limited set of hardware. (Although it increases my puzzlement as to why you are happy to lose performance and why you need to virtualize anything!)


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 1:49 am 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
iansjack wrote:
SoulofDeity wrote:
iansjack wrote:
It seems significant to me that almost every example that you come up with relates to emulating gaming consoles.

That's because that's what I'm talking about.

OK, that makes things a lot clearer. I hadn't realized that this was an OS aimed at games consoles. That makes life a lot easier as you are supporting a limited set of hardware. (Although it increases my puzzlement as to why you are happy to lose performance and why you need to virtualize anything!)


When I said, "That's because that's what I'm talking about.", I was referring to emulating consoles. In specific, a very simplistic fake console. As I have said time and time again, there is no loss of performance when the virtual drivers are hooked by the VM.

Alright, let's put it this way. You download a game off steam which requires DirectX12 to run, and your graphics card only supports DirectX11. What do you do? Nothing. You're f*$^ed until you go out and buy a new graphics card. Even if 'technically speaking' the game only uses 1 function that is not available in DirectX11. For the VM I'm describing, the you'd just download the latest virtual driver and it works. All of the functions that your physical hardware supports will be used thanks to VM hooks. The ones that aren't will use the ones in the virtual driver.

Tell me, which would you prefer; not being able to use your software period unless you can shell out a few hundred dollars for a new graphics card, or to have the game run on any platform at nearly the same speed via hooks in which there is possibly no noticable difference if your pc is fast enough?

Take another case, the Sony PSM developer's kit uses a custom USB driver to transfer the applications to the developer's assistant on the console. This driver only runs on Windows. Would you prefer to have to go out and buy a copy of Windows (or pirate it if you'd if that's your thing), and install it either on your pc or a VM (in the later case, being prepared to pull your hair out trying to forward your guest networking and USB drivers to your host); or would you prefer being able to just run it and have it simply work (LLE emulation)?


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 1:56 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4594
Location: Chichester, UK
Sorry, I really have no particular interest in games consoles (or emulating them). I'm sure that your proposed OS will be a huge breakthrough in that market but it's not something that is relevant to me.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 2:05 am 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
iansjack wrote:
Sorry, I really have no particular interest in games consoles (or emulating them). I'm sure that your proposed OS will be a huge breakthrough in that market but it's not something that is relevant to me.


(facepalm) It has nothing to do with games, it has to do with compatibility. All software written for the VM just works. No exceptions.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 3:16 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4594
Location: Chichester, UK
SoulofDeity wrote:
iansjack wrote:
It seems significant to me that almost every example that you come up with relates to emulating gaming consoles.

That's because that's what I'm talking about.
SoulofDeity wrote:
iansjack wrote:
Sorry, I really have no particular interest in games consoles (or emulating them). I'm sure that your proposed OS will be a huge breakthrough in that market but it's not something that is relevant to me.

(facepalm) It has nothing to do with games

I'm sure that you know what you are talking about and can reconcile those (apparently) contradictory statements. But I'm far too confused to continue this conversation.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 10:49 am 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
iansjack wrote:
SoulofDeity wrote:
iansjack wrote:
It seems significant to me that almost every example that you come up with relates to emulating gaming consoles.

That's because that's what I'm talking about.
SoulofDeity wrote:
iansjack wrote:
Sorry, I really have no particular interest in games consoles (or emulating them). I'm sure that your proposed OS will be a huge breakthrough in that market but it's not something that is relevant to me.

(facepalm) It has nothing to do with games

I'm sure that you know what you are talking about and can reconcile those (apparently) contradictory statements. But I'm far too confused to continue this conversation.


I'm sure that you're incapable of reading.

SoulofDeity wrote:
When I said, "That's because that's what I'm talking about.", I was referring to emulating consoles. In specific, a very simplistic fake console.


I'm not in the mood to bicker with you again. Let's stay on topic. Oh, and by the way, since it's completely obvious you didn't read even the first sentence, I'll quote the most important part of the post:
SoulofDeity wrote:
Alright, let's put it this way. You download a game off steam which requires DirectX12 to run, and your graphics card only supports DirectX11. What do you do? Nothing. You're f*$^ed until you go out and buy a new graphics card. Even if 'technically speaking' the game only uses 1 function that is not available in DirectX11. For the VM I'm describing, the you'd just download the latest virtual driver and it works. All of the functions that your physical hardware supports will be used thanks to VM hooks. The ones that aren't will use the ones in the virtual driver.

Tell me, which would you prefer; not being able to use your software period unless you can shell out a few hundred dollars for a new graphics card, or to have the game run on any platform at nearly the same speed via hooks in which there is possibly no noticable difference if your pc is fast enough?

Take another case, the Sony PSM developer's kit uses a custom USB driver to transfer the applications to the developer's assistant on the console. This driver only runs on Windows. Would you prefer to have to go out and buy a copy of Windows (or pirate it if you'd if that's your thing), and install it either on your pc or a VM (in the later case, being prepared to pull your hair out trying to forward your guest networking and USB drivers to your host); or would you prefer being able to just run it and have it simply work (LLE emulation)?


Last edited by SoulofDeity on Sat Mar 07, 2015 11:02 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 11:00 am 
Offline
Member
Member
User avatar

Joined: Sat Mar 31, 2012 3:07 am
Posts: 4594
Location: Chichester, UK
Quote:
I'm not in the mood to bicker with you again.
Agreed - this is a complete waste of time.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 11:40 am 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
Well, it seems we all agree now that this idea is completely possible and solves several compatibility and security problems that .NET and Java don't with a justifiable speed tradeoff for unrecognized virtual drivers.

Now it's a matter of deciding on how the virtual hardware works. The first thing I think I'll tackle is the graphics. Simply creating a framebuffer probably isn't a good solution, since that places a restraint on how large the resolution can be. I think a good solution would be to use a memory-mapped parallel bus. Then the resolution would be independant of the how much RAM the device has and the only real constraint is speed, which is dependent on the bus width.

For storage, I'm thinking of something along the lines of a peripheral interface. This could also probably be used for clipboard transfers or connecting to external devices.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 4:13 pm 
Offline
Member
Member
User avatar

Joined: Wed Jan 06, 2010 7:07 pm
Posts: 792
SoulofDeity wrote:
Tell me, which would you prefer; not being able to use your software period unless you can shell out a few hundred dollars for a new graphics card, or to have the game run on any platform at nearly the same speed via hooks in which there is possibly no noticable difference if your pc is fast enough?
I'll take door number three: have the software run on any platform at full speed via a well-designed, properly-versioned interface, that requires no hooks of any kind because software that uses a feature just uses it directly in the interface version that has it.

Your examples don't really convince me. DirectX12 in particular supports DirectX11 hardware, and back when GPUs changed a lot more DirectX did software emulation for that sort of thing anyway (or didn't bother because software emulation would be worthless, but that's not inherent in the way DirectX works- you can just not do it that way).

The Sony USB driver example is even less convincing. Sony wrote the driver for Windows, and creating a new platform (with locked-down hardware interfaces!) is not going to convince them to rewrite it. Any platform is going to have this problem, and your suggestions only make it worse.

_________________
[www.abubalay.com]


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 6:29 pm 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
Where did I ask Sony to rewrite their drivers? Where did I claim this is a gaming system? Where did I claim that DirectX12 can't run on DirectX11 hardware? Nowhere. These are examples. I'm seriously calling your reading comprehension skills into question at this point. If you can't understand the significant points of the examples I've given, how the hell are you able to understand a single word of the technical information used in OS development?

Man, it just pisses me off that all anyone here wants to do is argue...


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 7:10 pm 
Offline
Member
Member
User avatar

Joined: Wed Jan 06, 2010 7:07 pm
Posts: 792
I've explained exactly why I don't agree with your basic premise or your examples, none of which actually show the claimed benefits of your proposed design over the simpler design of a direct, versioned interface (which yours ends up including in an ad hoc way anyway). If you can't argue your case with examples that actually make your point, what conclusions am I supposed to draw?

On the other hand, if you could explain something your design makes possible, or does more easily, or more simply, or more efficiently, I promise I'd apply all of my pitiful reading comprehension skills to it.

_________________
[www.abubalay.com]


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Sat Mar 07, 2015 10:57 pm 
Offline
Member
Member

Joined: Wed Jan 11, 2012 6:10 pm
Posts: 193
Rusky wrote:
I've explained exactly why I don't agree with your basic premise or your examples, none of which actually show the claimed benefits of your proposed design over the simpler design of a direct, versioned interface (which yours ends up including in an ad hoc way anyway). If you can't argue your case with examples that actually make your point, what conclusions am I supposed to draw?

On the other hand, if you could explain something your design makes possible, or does more easily, or more simply, or more efficiently, I promise I'd apply all of my pitiful reading comprehension skills to it.


I have explained it. Like 50 times. With many examples of how it works, why it works, examples of it working in other situations, the benefits, the drawbacks, and comparisons to other alternative implementations. It's not my fault if you can't read, choose not to read, or don't understand what you're reading. I'm done spoonfeeding.


The How and Why
Quote:
#1
I don't think I explained it well enough, so I'll take another stab at it. When I say, "VM" I mean literally, a Virtual Machine or console. The machine will have simulated virtual hardware. The drivers and libraries for this VM will be implemented in software for the VM itself. The libraries may change over time, but the (virtual) hardware will not. (Physical) hardware acceleration would work via HLE hooks. If the libraries update, the user just has to download the latest version of the library. But regardless, anything written form the VM will work on any version of the VM for any platform without question.

#2
VM software cannot ask for resources outside of the VM, but resources that are outside of the VM can poke into it with HLE hooks. The VM software has no way whatsoever of knowing about these hooks. They are totally ignorant of anything outside the VM.

#3
No. Alright, look at it like this. The (virtual) hardware give it's (virtual) drivers access to a (virtual) framebuffer. The (virtual) driver implements a software 3D renderer. Other libraries use this driver. The VM sees this driver and patches it with HLE hooks that redirect to (physical) driver functions when the code is recompiled. The VM software only knows and cares about it's virtual hardware and drivers. These work 100% standalone of the physical hardware or platform because it's all software implemented. Because it uses virtual hardware, people can write their own drivers. However, they won't run as fast unless they are recognized and patched by the VM.

#4
The virtual hardware has absolutely nothing to do with performance. It simply provides a standard interface for the virtual drivers. What they decide to do with it is up to them. If they want to add a 'drawKitten' function, that's their prerogitive. The job of the VM is: "do I recognize this driver? if yes, do I recognize the function? if yes, are there any functions in the physical drivers that could be used to provide hardware acceleration for this function? If yes, patch a HLE hook for the 'drawKitten" function to provide hardware acceleration for it. If no to any of these questions, just use normal LLE emulation."

#5
The user applications and/or their libraries for the VM rely on the virtual drivers. These drivers are for virtual hardware that never changes, but they themselves can be updated. When the virtual hardware is simulated, this is Low-Level-Emulation, and all drivers are guaranteed to work exactly how they were written. Simulating hardware is slow though. .NET and Java sidestep this issue by not using virtual hardware and simply allowing the software to run native code. This makes the software both hardware and platform dependent.

My solution is to not allow anything running inside the VM (including the virtual drivers, libraries, and applications) to run any native code or probe information about the physical hardware or platform in any way. They are blind, deaf, and numb to anything outside of the VM. However, the VM sees that a driver called 'opengl.drv' is being uses by a program or library. It checks if it recognizes the driver by comparing either a checksum or globally unique identifier. If it recognizes it, it iterates through a list of all the functions in the driver.

Suppose it sees, "glDrawArrays". It checks the list of hooks it has for 'opengl.drv' and notices it has a hook for that. Next, it checks if the physical hardware drivers support that hook. If so, then when it recompiles the code that executes this function (remember, this is a VM) it uses High Level Emulation; substituting the call to 'glDrawArrays' with one to the physical driver. The software running in the VM does not know this is happening. 'glDrawArrays' still looks the same in memory, is at the same location, has the same size, etc. It has no clue whatsoever that anything has changed. The only difference is that the function is magically running either faster or slower than normal.

#6
The interface is simply a bare minimum for the drivers to build off of. Take for example, the TI-83+. It literally has no direct graphical hardware interface aside from a port that can be used to select a row and column, then either read or write 8-bits representing 8 pixels in a horizontal line. What did people do with this? Well, first they wrote a fastcopy routine that could blit an entire block of memory to the screen representing a framebuffer. Next, they wrote putsprite routines that would render to the framebuffer by either simple copying, xor-masking, or using 2 sprites (1 and-mask to clear space and another to blit it with or-ing). But it didn't stop there. Someone else calculated the frequency of the screen and determined exactly how long it takes for the juice to leave it. Then, they created an interrupt routine that would apply a checkered mask between 2 or more framebuffers; allowing 3 to 11-level grayscale on a monochrome screen. Someone else wrote a routine that could render skewed or rotated sprites. Then another person wrote a 3D renderer with texturing. From nothing but simple port io, you can now play DOOM on a TI-83+ or watch grayscale movies.

The VM just says, "oh, I see you're doing fastcopy, setpixel, putsprite, putrotatedsprite, and drawarrays. the physical drivers can do those hella faster, so I'll just use those instead. You'll never know the difference."



Benefits, Drawbacks, and Comparison to alternative implementations
Quote:
#1
What I'm trying to explain is that the drivers for the VM don't have anything to do with the OS. The VM is a thick iron wall between the systems applications and user applications. The user applications implement their own sandboxed drivers for the virtual hardware which are guaranteed to work, and the VM decides whether to execute these via LLE (slow), or use HLE hooks that translate those functions into functions for the physical drivers. At the very worst scenario, the software will run slow; at the very best scenario, all of the virtual driver functions translate directly to physical driver functions and the VM software is just as fast as .NET or Java. In either case, the software is 100% guaranteed to work regardless of the hardware or platform. This is something that can't be said for the other two because they allow native code execution.

#2
All software is guaranteed to work on all platforms and hardware combinations. Period. The only factor is how fast they will be executed, and that's determined by both the physical drivers and the VM.

#3
The entire point of this idea was to provide a better (and portable) environment for user applications, born out of frustration of cases when developers do stupid things like making their software only work on NVidia cards, or use their own proprietary drivers for things that don't work on all machines.

#4
There is no guessing involved. The VM just uses a UID or checksum to identify the driver. The magic is that even if the VM doesn't recognize the virtual driver, IT STILL WORKS! Why? Because the virtual hardware never changes, and thus it can fall back on LLE. It'll be slow, sure. At least until someone creates an HLE hook for it. Then it's just as fast as if you're directly using the physical hardware.

#5
Alright, let's put it this way. You download a game off steam which requires DirectX12 to run, and your graphics card only supports DirectX11. What do you do? Nothing. You're f*$^ed until you go out and buy a new graphics card. Even if 'technically speaking' the game only uses 1 function that is not available in DirectX11. For the VM I'm describing, the you'd just download the latest virtual driver and it works. All of the functions that your physical hardware supports will be used thanks to VM hooks. The ones that aren't will use the ones in the virtual driver.

Tell me, which would you prefer; not being able to use your software period unless you can shell out a few hundred dollars for a new graphics card, or to have the game run on any platform at nearly the same speed via hooks in which there is possibly no noticable difference if your pc is fast enough?

#6
Take another case, the Sony PSM developer's kit uses a custom USB driver to transfer the applications to the developer's assistant on the console. This driver only runs on Windows. Would you prefer to have to go out and buy a copy of Windows (or pirate it if you'd if that's your thing), and install it either on your pc or a VM (in the later case, being prepared to pull your hair out trying to forward your guest networking and USB drivers to your host); or would you prefer being able to just run it and have it simply work (LLE emulation)?


Examples of it working
Quote:
#1
Think of it as how an emulator works. When it begins executing a rom, the first thing it does is search through the executable code and apply patches to hook OS calls. The code in the rom itself doesn't know this is happening. All it knows is that 'osLoadProgram' loads a program. It just executes faster or slower for some magical unknown reason.

#2
This idea is nothing new. Emulators do this all the time. For example, Nintendo64 emulators do a checksum of the microcode for the RSP when it's loaded to determine whether to provide HLE for certain instruction sets like Fast3D or F3DEX2. The virtual hardware has nothing to do with performance, all it does is abstract away the physical hardware so that all drivers are consistent. They work. No ifs, ands, or buts. How well they work is dependant on whether the VM decides to use LLE (slow) or HLE (fast).

#3
No, the one who's missing the point is you. The VM doesn't accelerate hardware, it accelerates software. Perhaps it'd be more clear if instead of saying "virtual drivers", I said "microcodes". Take for example, the N64. It had no hardware support for shadows, and only a 1-bit stencil buffer. The guys at Rare wrote their own microcode that allowed them to do this and have crisper graphics, thus extending the functionality of the N64 (this is actually somewhat surprising considering that there were no tools to do so. they actually reverse engineered it and made their own tools)

Rather than emulating the hardware that executes the microcodes, most emulators emulate what the instructions in the microcodes do.


Top
 Profile  
 
 Post subject: Re: Design Choices of my OS
PostPosted: Mon Mar 09, 2015 5:33 am 
Offline
Member
Member
User avatar

Joined: Wed Oct 18, 2006 3:45 am
Posts: 9301
Location: On the balcony, where I can actually keep 1½m distance
SoulOfDeity wrote:
I have explained it. Like 50 times
That alone guarantees you're going at it the wrong way.

People disagree. People give counterarguments. People point out their opinions. Critics are always the loudest. The moment you believe your opinion is the only right opinion you started a religion instead of a discussion. And like pretty much any other uncritical followers of any religion, you get additional spanking for doing so on a scientific forum.

What all your critics are pointing out - besides the brokenness by choosing comparisons too quickly - is the fact that your design works on paper doesn't preclude it from being the next itanium and fail regardless of apparent merit.

_________________
"Certainly avoid yourself. He is a newbie and might not realize it. You'll hate his code deeply a few years down the road." - Sortie
[ My OS ] [ VDisk/SFS ]


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 60 posts ]  Go to page Previous  1, 2, 3, 4

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 20 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group