Actually, drivers aren't what I'm worried about. If I have some basic support for keyboard, mouse, storage, network and sound (which I do on one laptop at least), that's all that I really need from the hardware before I could call it somewhat usable. Having an actually useful userspace is the hard part, especially if you don't want to be fully POSIX. vim/bash/gcc/make is nice, but actually useful is a broswer, mail client, etc.
For device drivers, the most important thing is the interfaces and functionality that drivers depend on. Beyond that you can get by with some generic drivers (AHCI, EHCI, "raw framebuffer", HD audio, etc) and hope that eventually one day volunteers write drivers for the OS.
For "user-space without porting stuff"; I'm expecting to spend a decade on developer tools (IDE, compiler, etc) before anyone is able to write a simple "Hello world" application.