EDITORS’ UPDATE, Oct. 22, 2003
— A correction has been made to Martin Taylor’s response to the seventh question below, regarding study findings that Microsoft Windows Server 2003 outperformed Linux mainframes on standard file-sharing and Web serving benchmarks by 20 to 300 percent.
Martin Taylor, General Manager, Platform Strategy.
REDMOND, Wash., Sept. 15, 2003 — Now more than ever, businesses are looking for substantive data to help them quantify IT costs and determine the overall value of their IT investments in an effort to watch their bottom line. This concern has raised questions about software alternatives to Microsoft. To help address customers’ questions about performance and cost when comparing Microsoft products to Linux-based offerings, and to give them third-party comparative data, Microsoft commissioned two new pieces of research.
Microsoft asked Giga Research, now a wholly owned subsidiary of Forrester Research Inc., to run a detailed, objective comparison and analysis of portal application development and deployment for .NET on Microsoft Windows and J2EE on Linux for medium and large-sized companies. Microsoft also commissioned an independent third-party performance benchmark, audited by Meta Group, to compare server consolidation using Microsoft Windows Server 2003 versus a Linux-based IBM mainframe. (See links to both studies at right.)
Giga’s findings indicate that, as a portal application development platform, running .NET on Windows offers substantial cost savings over Linux and the J2EE development environment — as much as 25 to 28 percent less over a four-year period. Additionally, the IBM/Linux benchmark indicates that an Intel server running Windows Server 2003 performs significantly better — 20 to 300 percent better — than an IBM zSeries 900 mainframe running Linux.
PressPass spoke with Martin Taylor , Microsoft’s general manager of platform strategy, about the studies and the significance of the findings for customers, and with John Rymer , vice president at Forrester Research, regarding the methodology of Forrester’s cost-comparison study.
PressPass: Microsoft commissioned Giga to employ its Total Economic Impact (TEI) methodology and conduct a cost-comparison study between .NET on Windows and the J2EE/Linux platform for portal application development. The company also initiated an IBM/Linux performance benchmark. Can you explain the circumstances of each and what motivated Microsoft to initiate the research?
Taylor: After we released results of IDC’s total cost of ownership (TCO) study in December 2002, our customers asked us for additional credible, evaluative data that would help them make value- and performance-based IT decisions. The Microsoft-commissioned IDC TCO study suggested that Windows Server 2000 offers a lower TCO when compared to a Linux solution across four of five common IT workloads. The two most recent research projects, the results of which were released last week, reflect Microsoft’s continued commitment to provide customers with the concrete data they need to make strategic business decisions.
Microsoft wanted to help customers better understand how operating-system acquisition costs fit within the larger context of their IT investments. So we asked Giga to conduct a TEI study and compare the total economic impact of developing and deploying portal applications using .NET with Windows versus J2EE/Linux. The findings help to move the industry discussion from a focus on the cost of OS acquisitions and the perceived “free” price of Linux to a much more comprehensive look at the big picture — taking into consideration the flexibility, benefits and overall risk of the operating system.
In the same drive for comparative evidence, Microsoft commissioned a standard performance benchmark, conducted by third-party mainframe Linux specialists, to provide customers with concrete, objective evidence that either supports or refutes IBM’s recent, unsubstantiated assertions about the superiority of running Linux on the z900 mainframe. These claims have included the ability to run hundreds of Linux images and to consolidate the file, print and Web-serving workloads from hundreds of Windows NT-based servers. Until this study, no data that supported or refuted these claims was available for customers — not even from IBM. Conducting such a benchmark requires very specialized skills and is an expensive proposition, but we decided that it was worth the cost of carrying out the benchmark, and of having the results audited, to provide performance data that could help customers make better informed decisions.
PressPass: Can you provide some details on the Total Economic Impact methodology that Giga used in the cost-comparison study?
Rymer: Giga’s TEI methodology is designed to capture and properly communicate the value of IT initiatives. In so doing, TEI considers four aspects of the initiative: the costs, which are sometimes referred to as TCO, as well as the benefits, risks and flexibility. Giga defined a model for an average portal application using information drawn from in-depth interviews with 12 organizations and then plugged product pricing and labor costs into the model to compare the costs associated with the two development options. The model represents an average case.
Giga interviewed 12 organizations from both the public and private sectors to obtain financial information about significant development projects conducted by those organizations. Seven of the subjects used primarily Microsoft’s platform and tools; five of the subjects used Linux and J2EE. We then created two TEI models one for a large and one for a medium-sized organization — for a typical portal application. We chose to focus on portal applications because these applications were most common among the interview subjects.
PressPass: What were the primary results of the Giga TEI study?
Rymer: The primary conclusion of the study is that Microsoft offers a substantial cost advantage over J2EE/Linux as a development platform for the applications considered. We found that the J2EE application server and Unix-based database software used in the Linux development and deployment stack drove up product costs and development complexity relative to the comparable Microsoft products. In contrast, Microsoft’s tools simplified development of applications like those profiled in the study when compared to the J2EE/Linux products. This simplification translates into lower labor costs for development and administration of custom applications and a faster time to deployment.
The comparison of the two platforms signifies that large-size to medium-size organizations that develop, deploy, support and maintain custom applications on the Microsoft .NET platform can expect to experience 25 percent to 28 percent less cost during a four-year life cycle than if they used the J2EE/Linux platform.
PressPass: How are these findings significant? For instance, how do application development costs square with the myriad factors that IT professionals must consider when making decisions about infrastructure implementations?
Taylor: The link between the J2EE environment and Linux is often overlooked in discussions of the cost of Linux. As I stated earlier, we believe this study helps move the industry discussion from a focus on OS acquisition to an examination of the total economic impact of an IT investment. It also demonstrates that for enterprise customers, Microsoft’s Windows platform costs less and offers better integration benefits than Linux plus commercial add-ons, such as J2EE.
While cost is only one of the factors that should be considered when making an IT system decision, it is a high-profile and significant one. Amidst an economic climate in which most companies are strongly focusing on their bottom lines, cost considerations have never been more top of mind. Giga’s TEI study findings show that the full development and deployment environment and the labor associated with it comprise the biggest costs when it comes to an organization’s application platform investment. It is in these critical areas that Microsoft’s integrated innovation approach to building software enables significant economic advantages in the form of greater developer productivity, faster time to market and greater ease of use — which all result in lower overall costs and higher value-adding benefits for an organization.
We hope to see enterprises take a closer look at the long-term business value they require from their IT solutions and Microsoft’s commitment to delivering such value to customers above and beyond Windows cost advantages.
PressPass: Can you provide some background on the other research that was published last week — the Linux/IBM performance benchmarks?
Taylor: Microsoft contracted with a third party experienced in running the SuSE operating system on Linux and IBM mainframes to carry out the benchmark. To accurately measure performance of the two competing platforms, Microsoft leased two months of dedicated access to two CPUs of an IBM z900 mainframe, which was the most recent model available — and the one that IBM referenced when making its claims about Linux/IBM superiority. The mainframe had 32 GB of memory installed, 24 GB of which was available to the two CPUs. The lab used SuSE Enterprise Linux 8.0 as the operating system, the VeriTest NetBench 7.03 benchmark against Samba for file serving, and the WebBench 4.1 benchmark against Apache for Web serving. With this dedicated access to the entire machine and Shark storage array, no other workloads or tasks were being run during the tests. Microsoft provided 96 PC clients and ran standard, reproducible test mixes from VeriTest, which included various numbers as well as combinations of virtual servers and clients.
To ensure validation of the study, Microsoft employed the Meta Group to audit the benchmark to determine if the tests complied with the procedures specified by VeriTest for carrying out the benchmarks. Meta Group visited the site to inspect the configuration and examine the test results. After determining that the benchmark complied with the procedures specified by VeriTest, Meta certified the benchmark results.
PressPass: How did you decide to use two CPUs in the study? Could you have used more?
Taylor: Using two CPUs gave us a way to look at scalability going from one to two CPUs. While we could have used more, the cost was extremely high and there was no reason to believe the results would have been significantly different. No computer architecture gives you better scalability going from two to three CPUs after going from one to two.
PressPass: In general, what were the results?
Taylor: The results directly refute the claims that IBM has been making about running Linux on the mainframe to consolidate Windows servers, especially for file serving and Web serving. IBM claims that one zSeries CPU can handle the workload of three to four Intel CPUs, that it can migrate 20 Windows NT file servers onto one Linux image, that hundreds of virtual machine images are possible with the zSeries/Virtual Machine platform, and that the Virtual Machine OS provides an environment that lets you create and manage hundreds to thousands of VM guests, in which multiple Linux images can be operated on a single S/390 or zSeries platform.
Rather than performing the work of three to four Intel CPUs, we found that each mainframe CPU performed 14 percent less work than one 900 MHz Intel Xeon processor running Windows Server 2003. The Windows results were from a comparable study performed earlier in the year by VeriTest and are available on their Web site. We also found that mainframe Linux performed poorly on standard file-sharing and Web-serving benchmarks — Windows Server 2003 performed between 20 and 300 percent better in the VeriTest study. Furthermore, for file and Web serving, the results do not support IBM’s claim that hundreds of Windows servers operating at 5 percent capacity can be consolidated to run on one or two z900 processors. Simply put, the results on the WebBench benchmark showed that Web server consolidation on a z900 running Linux makes no sense.
PressPass: Studies often prove what they set out to prove in the first place. Can you talk a little bit about why you believe this is an objective study?
Taylor: We feel confident that the results are objective because standard benchmarks were used, the tests are completely documented and reproducible, and because Microsoft employees were not involved in carrying out the benchmark. The lab followed the procedures specified in the VeriTest benchmarks and in IBM Redbooks for configuring mainframe Linux. In addition, the lab was staffed by experts in mainframe Linux and we have a high degree of confidence that they did everything they could to get the best possible performance. The META Group’s review of the benchmarks support the rigorous nature of the methodology we used.
PressPass: Can you comment on the performance of Microsoft Windows NT vs. Windows 2003 compared to Linux on the mainframe? What advice might you give to a company looking to consolidate their Windows NT servers?
Taylor: Windows NT 4.0, a product that was released seven years ago, produces NetBench file-sharing results 20 percent below those produced by mainframe Linux. Windows Server 2003 performed over 150 percent better than Windows NT 4.0 on the same hardware, which we think attests to the advances that have been made in Microsoft server technology over that period.
Customers looking to consolidate Windows NT or other servers should first consider doing so on Windows Server 2003. Performance, reliability, security, availability, and manageability are just some of the reasons Windows Server 2003 makes an excellent consolidation target. Migrating to Windows Server 2003 would be a much more cost effective way to reduce the number of servers in use than to try to consolidate onto a mainframe running Linux.
PressPass: Overall, what would you say is the general significance of the findings for both projects?
Taylor: Microsoft hopes that this latest research, in combination with the IDC TCO study conducted last year, provides customers with the sort of third-party data and information they need to make well-rounded, cost- and value-based IT decisions. We believe this set of evidence reinforces the value of Microsoft’s integrated innovation approach to building software, which enables significant economic advantages for application development and deployment in the form of greater developer productivity and faster time to market. In addition, Windows Server 2003 is shown to be a much more cost-effective solution in contrast to using a Linux mainframe solution to consolidate workloads. For customers looking to migrate and consolidate Windows NT 4.0 file, print, and Web workloads, these cost, productivity and performance advantages provide a compelling proposition for landing on Windows, as opposed to Linux, as they make this move.
With evidence like this, we wish to continue to address customers’ questions and help them quantify the value of their IT investment decisions.
