rdos wrote:
I actually fail to see how you effectively could test OSes with automated tests. Just getting it to build & boot is simply not enough for a real test. I always do testing on real hardware and not in emulators as I find that emulators often work even when real hardware wouldn't.
I totally agree that on VMs / emulators it's easy to get the stuff working. The real test is on the real HW. But, that doesn't stop you from having automated tests. Obviously, just getting to build & boot is nothing: there's much more that can be done. For example, you could test your syscalls in a variety of scenarios using a program running on your OS. A 100k LoC projects has an
incredible amount of code paths. Also, you can have kernel self-tests as well and trigger them with a special syscall interface. That will allow to test special code paths that are not easily triggerable from outside. And if that's still not enough, you can write multi-threaded stress/chaos tests that will expose any race conditions in your code. All of that can run both on VMs and on real hardware as well.
rdos wrote:
About the worse thing that can happen to an OS project is the introduction of random crashes. I've had a few of those were I had to revert code back 100s of revisions like when I introduced multicore operation, but also to a lesser extent when I modified the physical memory handler. I don't think automated testing would catch these scenarios. Often, lengthly tests on as many diffrent machines as possible is required.
Oh yeah, random crashes are terrible. Automated testing using CI helped me catch many of those cases. Actually, I started writing more tests just to increase the stability of my kernel. I observed that, even if on my local machine in a VM is harder to reproduce certain bugs, when the same tests are run in CI system in the "cloud", I was able to catch more bugs because there the my code was preempted much more often so, interrupts might come later etc. In other words, the opposite of a real time system. That's a terrible environment, but it's also good because it "shakes" the whole timing and triggers real bugs (race conditions).
Obviously, I'm talking about
pure software bugs. When you have bugs in your drivers related with
how you interact with real HW (which is much tricky to work with compared to the emulated devices), well, there you have no choice other than running your OS on the real machine and (possibly) trying to reproduce the problem there, with tests.
OK, just for completeness, I'll add that in my knowledge, big companies can do automated tests with hardware as well: they have special HW devices (assumed as reliable) that simulate real input (PS/2, USB, Ethernet, PCIe, whatever) so they can treat a whole machine (HW + software) like a sort of "black box", exactly as we can test an OS using a VM. Each of those special devices, looks like a regular device of type XYZ to the tested machine's kernel, but in reality, its behavior is controlled from outside. Not an expert here, I just know such things exist.