I settled on an interface where only the driver (in userspace, being a microkernel) has access to the frame buffer. Applications can ask the graphics driver to create a texture, which is a shared memory block between the driver and application, and the texture has a unique ID. Only the owner of a texture can draw into it. (At some point I need to create a permissions system on who can read other programs' textures.)
There is a special texture - texture 0 - that represents the framebuffer. Only one process at a time can have permission to draw straight into the frame buffer (this is going to be the window manager, but at some point I could also support full screen games.)
I also have a batch API, where I can string together a list of draw commands (currently only BitBlt operations between textures, which are implemented in software), and send them all at once as a single message to the graphics driver. The idea here is that you could string together a bunch of draw calls and maybe one day I'll support VMWare/QEMU's
SVGA-II, and the driver would compose the scene together. If a developer wanted, they could create one big texture and treat it as a pixel buffer and avoid the graphics driver's operations completely until it was time to BitBlt it to the screen.
Now I'm thinking about how I'm going to deal with the window manager. It will be a compositing window manager. Each window will have a texture associated with it. Programs are able to draw whatever they want into the textures (either via the graphics driver, or via shared memory.) It's up to the individual program if they want to double buffer (in fact, we could implement a kind of page flipping: draw into texture 4, tell the WM "Window is now texture 4", draw into texture 3, tell the WM "Window is now texture 3", repeat - it would require making the "Window is now texture x" send a response so we now know it's safe to draw into the other texture.)
I'd like implement this with no shared memory needing to exist between programs and the window manager (only the programs and the graphics driver, and the WM and the graphics driver would share memory.) The WM would know the texture IDs for each window, and have a texture that it draws the window decorations to, a texture containing the mouse, and in a single IPC, send one batch of draw calls with what has changed to the graphics driver.
The disadvantage of this method is that I can't do cool effects such as blurs or shadows, unless I have an draw call to do that (and then I could go down the rabbit hole of pixel shaders, which is overkill for a hobby OS) or I do multiple IPCs - e.g.:
- Window Manager tells Graphics Driver to draw the windows' textures into the WM's texture. (IPC 1)
- Graphics Driver tells Window Manager it is done. (IPC 2)
- Window Manager samples the WM's texture to do fancy shadows and blurs.
- Window Manager tells Graphics Driver to draw the WM's texture to the frame buffer. (IPC 3)
What have other people done with this graphics drivers and window managers?