REDMOND, Wash., March 26, 2001 — What do you do if you receive 30,000 customer-feedback reports in just two months?
If youre John Gray, you pay close attention. As a manager responsible for Windows releases at Microsoft, Gray and his team help run Microsofts “Whistler” Server beta-testing program. Gray knows from experience that a rigorous review by customers is critical before the software is finalized and sent to manufacturing later this year.
“Whistler,” the code name for the next version of the Microsoft Windows server operating system, represents the customer-driven evolution of Windows 2000. With Whistler, Microsoft is building on the reliability, manageability and scalability that customers value in Windows 2000, while offering support for new, high-performance 64-bit hardware architectures based on Intel Itanium processors.
Businesses of all sizes are deploying the Windows 2000 server family in record numbers. Windows 2000 servers recently reached the one-million-licenses-sold milestone, just 12 months after launch — 30 percent faster than Windows NT 4.0.
With the release of the Whistler beta 2 server family, Microsoft sets in motion one of the largest and most ambitious software testing programs in its history. Nearly 300,000 customers, partners, OEMs, developers and other testers will have access to the latest version of its new server operating system software over the next few days, and all of their comments will be reviewed before the product comes close to being considered complete.
“The beta testing program is our opportunity to take our server products to a wide audience of people who will help us tune the fit-and-finish,” explains David Thompson, vice president of the Server Product Group at Microsoft. “The input we receive from customers is absolutely critical to assuring that we build the product they need and want, and one that will perform as they expect.”
Microsoft asks its beta 2 review participants to actively install, test and report their results. “This is not an evaluation-based testing program,” Gray says. “We want active feedback on issues encountered and scenarios tested.”
Gray notes that responses in a beta-testing program typically include reports of problems encountered with hardware or software, device compatibility and issues of general usability. “Beta testers give us honest and straightforward opinions,
“he says.”
They dont hold back, and we appreciate their viewpoint.”
The 14-member team charged with sifting through beta-tester feedback expects to receive as many as 30,000 comments on Whistler servers in the next couple of months, according to team-leader Paul Donnelly. That compares to the approximately 20,000 reports the team fielded after the first beta release of Whistler desktop and server software.
“Where computer hardware and software are concerned, its an incredibly complicated world,” Donnelly notes. “And thats where the beta program is most valuable. It offers the opportunity to test interactions between every imaginable combination of computers — hardware and software — configurations that a single organization just doesnt have the resources to test.”
For their part, beta testers take their role seriously. Stephen Vickory of Greensboro, N.C., a systems engineer at Pomeroy Computer Resources, has beta tested Microsoft products for almost seven years. In his evaluation of the beta 1 version of Whistler, Vickory gave Microsoft valuable feedback about hardware compatibility and stability after installing the OS on several systems and running it with all the software he typically uses.
According to Vickory, the value of beta testing is clear. “When end users get the final product, its going to perform better for them and be more polished,” he says. “Microsoft will resolve the issues that I find, so customers wont encounter them.”
Vickory adds that being a beta tester is a unique experience. “Seeing something before its done and contributing ideas to Microsoft, then seeing those ideas implemented in the final product, makes you feel like you really made a difference,” he says.
Leen Snoek of ‘s-Hertogenbosch, the Netherlands, has been making a difference in Microsoft software since 1998. A general manager of planning and control at a bank by day, Snoek often spends some of his free time beta testing Microsoft products. Snoek has two test systems, but the main testing is done on his “daily-use machine,” so he can spot any issues quickly.
“Its a kind of stress-test environment in a real-life situation,” explains Snoek, who is testing two products in the Whistler server family.
Snoek has also been impressed by the resources available to testers and the overall level of professionalism in the program. “The beta support teams are very positive-thinking and supportive,” he says. “They really listen to their testers if you suggest things in a constructive way.”
A substantial amount of feedback on Microsoft server products also comes from the Joint Development Partner (JDP) program, a group of about two dozen large corporations that focus on early live deployment of Windows in lab or production environments.
“The JDP program provides real-world, enterprise-level testing by some of our best partners in a very close relationship,” says Gray, noting that JDP organizations often deploy hundreds of servers and thousands of clients.
Thompson says that the Whistler server products now in beta testing build on the solid foundation of the Windows 2000 Server family. “Weve spent the past year since the launch of Windows 2000 listening to customers tell us about their evolving infrastructure needs and how we can continue to improve our products to meet those needs,” Thompson says. “Now well spend the next couple of months paying close attention to the feedback that we get during this phase of the Whistler beta testing cycle, as we draw the net tighter on finalizing the features and product configurations.”
To encourage as much customer input as possible, Whistler test versions are being distributed to a broad audience. The initial preview version of the OS went to technical beta testers, including independent software and hardware vendors whose participation helps assure third-party compatibility. The beta 1 release added more corporate customers and incorporated Microsoft Developer Network (MSDN) subscribers. The beta 2 version is going out to technical beta testers, MSDN developers and the 150,000 largely customer-IT professionals who make up Microsofts Technet.
“Not until weve sifted all the feedback from testers will Whistler be ready for final beta and release-candidate reviews,” Paul Donnelly emphasizes.
Beta testers submit their input using an online reporting tool thats included with their beta version of the OS. Reports, including information about the testers hardware system and associated software. The input is then sent over the Internet to a Microsoft database, and all comments about the same issue are routinely tied together.
Donnelly explains that responses from beta testers are prioritized as they arrive and addressed in order of importance. To set priorities, the team works to accurately assess the overall impact of any reported problem — namely, its severity and how many people might be affected. The team also determines whether a reported issue has already come to light and possibly even been resolved. In addition, the team looks at how many other reports have been received about the same issue.
“In this process, our team pores through
hundreds
of known issues, as well as scheduled-to-be-fixed and already-fixed issues,” Donnelly explains. “We work very closely as a team and tap each others knowledge to understand the problem. We also meet regularly with other teams responsible for certain areas of the OS to make them aware of what were seeing. This way, we can make certain theres an ever-increasing level of quality, and be confident that the changes we make are in fact addressing quality issues and increasing customer satisfaction.”
Test reports are made available to the product development team continuously, where refinements to the Whistler server family software code help ensure that the eventual product will be truly customer-ready come launch time.
Microsofts extensive beta testing and product-refinement process assures customers that the Whistler software they ultimately install on their servers will build on and enhance the core values of the Windows 2000 server family.
“Microsofts customer-centric model for building better software translates to clear customer benefits in the end,” Thompson says. “Its the heart and soul of making something thats solidly reliable, easy to manage, and able to be deployed so quickly that customers can get their ideas to market ahead of competitors.
“Giving customers competitive advantage. Thats what its about.”
