It’s so simple, on the face of it.
Mindcraft Inc., in a new study [http://www.mindcraft.com/whitepapers/nts4rhlinux.html] commissioned by Microsoft Corp., found that “Microsoft Windows NT Server 4.0 is 2.5 times faster than Linux as a File Server and 3.7 times faster as a Web Server.”
In reality, however, it’s not simple at all.
Mindcraft, an independent test lab, pitted the two network operating systems against each other on a Dell Computer Corp. PowerEdge 6300/400, testing both NT 4.0 Service Pack 4 and Red Hat Software Inc.’s Linux 5.2 upgraded to the Linux 2.2.2 kernel. This quad 400MHz Xeon SMP (symmetric multiprocessor system) had a whopping 4GB of memory (a gigabyte of which was used in the test) and a PowerEdge RAID II Adapter, 32MB cache set to RAID 0. The tests used ZD’s own NetBench and WebBench benchmark tests for file and Web serving tests.
As tested, NT beat Linux like a drum. But, that’s not the whole story.
Singing the right tune
Mindcraft tested NT with NT tuning, benchmarking and technical support from Microsoft, and Internet Information Server 4.0 tuning information from the Standard Performance Evaluation Corp. In the case of Linux, though, Mindcraft President Bruce Weiner was “surprised that Red Hat Software didn’t help us in tuning the OS.”
Red Hat spokeswoman Melissa London explained that “there was a query to our contract technical support, but they didn’t know what to do with Mindcraft’s request. Had the question came through the right channels — the public relations or the engineering staff — we’re sure Linux would have at least have had a better representation.”
In the future, Red Hat “would love to repeat the tests with our input, support and involvement,” London added.
In addition, Mindcraft said it found “little real tuning information in the Linux knowledge bases.” What information they did find was scattered over the Web. And for Mindcraft, at least, the “word of mouth” Linux support of newsgroups, mailing lists and online conference sites didn’t work.
That said, Jeremy Allison, a leading member of the Samba Team, an open source consortium, checked his records and could find no trace of anyone from Mindcraft asking in Samba’s public forums about Samba tuning. Indeed, after looking over Mindcraft’s published Linux parameters, he said, “[Mindcraft] didn’t do any tuning on the Linux side.”
Consequently, Mindcraft has found itself buried in Linux tuning “suggestions.” Many of these are just flames from outraged Linux users, but many of them, as Weiner admits, are well-founded.
For example, Mindcraft turned the Samba parameter ‘widelinks’ off. But, according to Andrew Trigell, the Samba Team leader, doing so “lowers the performance enormously. It adds 3 chdir() and 3 getwd() calls to every filename lookup. That will especially hurt any SMP system.”
Mindcraft’s Weiner agrees: “Clearly, we wouldn’t make that tune if we were doing that test today.”
The result? Highly tuned NT beats a barely tuned (if that) Linux. But, there’s more to it than that.
For one thing, Linux tuning information is hard to gather. It’s there, but it really requires an expert Linux user to find it. If Linux is to win commercially, information known only to Linux gurus and hidden away in a dozen Web sites needs to be gathered together and made more readable and approachable. For new users, learning advanced Linux is still much too difficult.
Take the Apache server, for example. There are several files on the Apache Project’s Web site on how to tune Apache and Linux. Even so, these files aren’t easy to find. On the flip side, finding information on tuning Internet Information Server 4.0 is easy at Microsoft’s knowledgebase.
Beyond tuning: incompatibilities
For Linux, SMP is a new option, incorporated in the 2.2 release. This is not the current base OS release that Red Hat is shipping today or that was used in the test. Instead, the system was upgraded to 2.2 and SMP activated by inexperienced Linux administrators using early Red Hat documentation. Moreover, the Linux RAID driver was a beta release and not well tested with the “so new the shiny hasn’t worn off” 2.2 kernel.
As a source close to Ziff-Davis Benchmark Operations, the ZD benchmark’s authoring agency, said of Mindcraft’s testing, “A more accurate headline might have been Linux does not scale up” instead of “Windows NT Server Outperforms Linux.”
Sm@rt Reseller’s own tests, using the same benchmarks, on low-end uniprocessor machines, has shown that Linux combined with Apache and Samba beats NT and IIS handily. (See Linux Up Close: Time To Switch and Samba 2.0: A License to Kill NT).
That said, it’s likely that a highly tuned NT, on the Mindcraft SMP platform, would prove a bear for even a tuned Linux to defeat. For NT, this kind of machine is a home field. For Linux, at this point in its development, it’s an away field — far, far from home.
On single-processor systems, Linux wins. On SMP/RAID systems, Linux is just now getting in the game. Even so, even at the high end of stress testing, Linux’s numbers would have been much closer to NT — if not higher — had Linux been as well tuned as NT.
A version of this story was first published in Sm@rt Reseller.