Messages in this thread | | | Date | Sun, 04 Jul 1999 11:50:54 -0700 | From | Miles Lane <> | Subject | Re: Are there kernel testing suites out there? We need them. |
| |
Rik van Riel wrote: > > On Wed, 30 Jun 1999, Miles Lane wrote: > > > One of the things Micro$oft does is stress test Windows NT after > > every build. > > Looking at the quality of Windows NT should be enough to establish > the usefulness of such a test...
I really must disagree with you on this. Granted, there are bugs in Windows NT. Many on these bugs are related to the legacy support of the 16-bit "DOS/Win3.1 compatibility" subsystem, plus the complexity and fragility of migrating what was formerly subsystem code into the microkernel. At the inception of NT, the main subsystem that hand-tuned assembly code that ran in ring 0 was the Networking Subsystem (please forgive me if my details are a bit off, it's been years since I was an NT network subsystem tester). My understanding is that more and more NT subsystem code has been allowed to bypass the subsystem architecture and been integrated into the microkernel. This has been done to decrease the size of the OS in working memory, to increase performance and decrease load time. This is done at the cost of portability (there's more assembly code) and kernel stability (if you screw up in the microkernel, you're system is hosed).
Anyhow, I can say from firsthand experience that Micro$oft's testing systems flush out a lot of bugs. The number of bugs that are seem by the public are a vanishing fraction of the ones Micro$oft has fixed or decided to leave in because of launch date requirements and bug severity/priority. Testing is a really good thing. So is easily reproducible regression testing (testing stuff that was broken in the past).
I'm not sure how much testing methodology gets discussed in this forum. There are many forms of testing:
API "white box" Application "black box" Stress Boundary condition Usability Production "burn-in"
My impression is that a lot of the Linux kernel testing is performed on an ad hoc basis by Linux kernel developers. Of course, early adopters of development kernels find many problems, but from what I've seen of this list, many folks who should know how to generate useful OOPS to report, don't.
Anyhow, using the instability of NT as an excuse to not develop systems to automate and simplify testing is just silly. Testing is the right thing to do. The question is what test systems would be helpful and feasible in the context of the OSS distributed development model.
I would argue that developing a test script repository that held tests for the various subsystems (video, networking, disk, and so on) sorted by CPU, kernel revision, distribution hardware, and so on, would be useful.
The primary benefit of such a system would be that less knowledgable or skilled development kernel users could help find bugs by stressing/testing their system in many ways.
I understand that Linux gets a lot of use, and so many bugs get found. But systematic methods of finding new bugs is a good thing.
> Stresstesting a Linux kernel on a known-good piece of hardware > will only show up the most obvious bugs and not the typical > race conditions and marginal-hardware cases that bother us most. > > Rik -- Open Source: you deserve to be in control of your data. > +-------------------------------------------------------------------+ > | Le Reseau netwerksystemen BV: http://www.reseau.nl/ | > | Linux Memory Management site: http://www.linux.eu.org/Linux-MM/ | > | Nederlandse Linux documentatie: http://www.nl.linux.org/ | > +-------------------------------------------------------------------+
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |