Q&A: Microsoft Improves Certification for IT Professionals with Performance-Based Testing

REDMOND, Wash., Feb. 1, 2005 — The Microsoft Certified Professional (MCP) program is one of the most successful certification programs in the industry with over 1.7 million individuals credentialed all over the world. The MCP credential is for professionals who have the skills to successfully implement a Microsoft product or technology as part of a business solution in an organization.

Microsoft wants to offer certification that helps assure MCPs are experienced and qualified to help with their IT problems. But traditional multiple choice tests don’t always do a complete job of testing IT professionals’ ability to perform practical, real-world tasks end to end. So the Microsoft Learning team has developed a new system of testing that evaluates on-the-job skills of an IT pro.

Reinforcing its commitment to the MCP community, Microsoft is about to incorporate performance-based testing in all core Windows 2003 exams. Through simulated environments, IT professionals taking exams for certification will now be required to demonstrate their knowledge of products and processes they may encounter on the job. This dramatic change in the testing protocol is designed to increase the value and credibility of the MCP credential.

To learn about the new testing methodology, PressPass spoke with Al Valvano, Lead Product Manager with Microsoft Learning, and representatives of the two exam delivery partners participating in the program – psychometrician Paul Jones, an expert on testing with Thomson Prometric, and Randy Trask, VP of Market Development for Pearson VUE.

PressPass: What is performance-based testing?

Valvano: It’s testing by doing. That’s the definition used by the Performance Testing Council, and it’s a good one. It tests for an individual’s ability to perform a task as opposed to recalling information. Think of it in terms of driving a car. If everybody who got their driver’s license only had to take a knowledge-based written test without the behind-the-wheel road test component, they would not be proving their ability to drive a car. The driving test adds validity to the written exam.

Training in IT today often employs lab-based exercises to reinforce learning. It’s evolved to the point where we use a lot of simulations. And now we want to insure that when an individual takes an exam to be certified in a particular technology, they’re being tested by doing. It’s a validation of their ability to accomplish given tasks.

Jones: With only multiple-choice items in an exam you can’t adequately address certain constructs, which is the knowledge in the person’s head that you’re trying to assess. You leave out components of the ability that you’re trying to measure. For that reason, simulation nicely complements the multiple-choice exams.

In multiple-choice you have four or five items from which to choose. But in a performance-based item, you have free play in an environment that works like the real world. You solve a problem and you show how to get from a starting point and navigate through a process. In addition, you must show how to manage the whole problem-solving effort and demonstrate the results. What performance-based testing adds is the components of self-management, monitoring, problem solving and planning a solution – which gives the testing program enhanced validity.

PressPass: Why is Microsoft incorporating performance-based testing (simulation) into its certification program now?

Valvano: Because we’ve found that performance-based testing produces more skilled candidates. And we’re hearing from both Microsoft Certified Professionals and IT hiring managers that there is greater benefit to them in performance-based testing over traditional testing. We’re always looking for ways to add value for our customers and including simulations in our exams increases the value of certification to the MCP community.

Testing with simulations raises the bar by requiring candidates to actually perform tasks to accomplish a specific goal. Candidates have to prepare for certification by really using the products. Memorization won’t cut it – you’ve got to develop the necessary skills. And that will help differentiate you from other professionals and improve your career opportunities. Hiring managers gain confidence that our certifications mean that potential employees can do the jobs they need them to do.

Jones: The good thing about testing with simulations is that it stimulates candidates to understand how to do things. And if that’s your focus as a student, your preparation is going to help you in the workplace. Exam preparation is more appropriate to what you’re trying to learn. In the measurement literature, that’s sometimes called “systemic validity.” That is, it’s pushing the whole certification system in the right direction. Candidates are focusing on the right kind of training and preparation.

PressPass: Is Microsoft taking any kind of unique approach to performance-based testing?

Valvano: We’re using a very robust simulation technology. Most other programs that use simulations as an integral part of their testing experience are for high-end certifications. They’re typically live applications. In other words, you come into a lab and you’re given a server and some hardware and specific problems to solve. You’re probably evaluated by an observer as well as with some automated scoring.

The problem with live application testing is that has limited scale. It’s very difficult to roll out on a global basis. So you’re limiting your audience. The Microsoft certification program has much greater reach, with thousands of people taking exams every single day all around the world. Now simulations will be available to everybody that can currently take our exams and they’re offered in eight languages: English, simplified Chinese, French, Spanish, German, Korean, Japanese and Brazilian Portuguese. Everybody who wants to attain a Microsoft certification will have the opportunity to have the same innovative testing experience.

PressPass: Where do people go to take the certification exams?

Valvano: Customers can take the exams from either of two exam delivery partners, Pearson VUE and Thomson Prometric. The simulation exams are going to be available at more than 6,000 test centers around the world. Implementing simulations into exams that are globally available has proven to be a challenging endeavor for us. We’ve made the commitment to performance-based testing and working with our partners, we’ve assembled an incredible team to do it.

Jones: It’s no cakewalk to do performance testing. It’s more complicated to develop and more expensive in every way. It sounds really great to people who have never done it but any company that is implementing performance testing technology wants to be sure that it’s being targeted at a place where there’s going to be a good return on investment.

PressPass: How are you getting the word out to IT professionals about the enhanced testing program?

Valvano: A page will be posted for our customers on the Microsoft Learning Web site describing the simulations and the exam technology. In March, they can also access a demo of the simulations and preparation guidance. The information is also going to be on the secure MCP Web site that is only available to those who are credentialed and in the monthly MCP newsletter. And then there are the thousands of customers currently going to Certified Partners for Learning Solutions (CPLS) for training and certification. We’ve already communicated with the CPLS network about our testing improvements and provided them with resources including videos to talk about simulations with their customers.

PressPass: Which tests incorporate simulations?

Valvano: We’re initially introducing simulations in the core exams for both our Microsoft Certified Systems Engineer (MCSE) and Microsoft Certified Systems Administrator (MCSA) certifications. These two exams – 70-290 Managing and Maintaining a Microsoft Windows Server 2003 Environment and 70-291 Implementing, Managing and Maintaining a Microsoft Windows Server 2003 Network Infrastructure – will be offered with simulations by March 31, 2005. Simulations will launch in many other exams throughout 2005. Look for a release schedule on the Microsoft Certification Web page in March.

PressPass: Is there any incentive for existing MCPs and other experienced IT professionals to take the exams with simulations?

Valvano: MCPs who have already taken a non-simulated version of 70-290 or 70-291 are not required to take the revised test but they may choose to refresh their credentials with the performance-based criteria. Retaking the exam with simulations can help an MCP refresh his skills and perhaps make him a bit more marketable.

PressPass: If a person has already completed some of the exams leading to certification, do they need to retake a version of the test with simulations?

Valvano: If they have passed 70-290 or 70-291 by the launch date of March 31, they don’t need to retake the exams. But if they have not yet taken these exams they must take the version with simulations. That’s the only form of the test that will be available after launch.

PressPass: What’s the best way to prepare for a performance-based exam?

Valvano: First, it’s important to note that we will still have a range of item types to measure the full spectrum of a subject: case studies, multiple choice, hot area, drag and drop, and others that an MCP must know thoroughly – so hitting the books is still something test takers will want to do.

For the performance-based items, we suggest looking at the “Skills Measured by this Exam” section of the exam prep guidelines. Make sure that you can perform the tasks noted in the objectives such as “Monitor, manage, and troubleshoot access to files and folders.”

Simulations will generally be associated with objectives that require the candidate to perform a specific task. Spend time in the environment setting up situations that are called out in the exam objectives. Lots of interaction in the environment will significantly contribute to a candidate’s ability to understand the environment and accomplish specified tasks. So practice, practice, practice!

Trask: Certification exams have always been about demonstrating that you have what it takes to perform a job. This really means that once certified, you should be able to hit the ground running in a new job. If you have the confidence you can get behind a computer and get the job done – you’re prepared, because that is exactly what Microsoft is going to ask you to do!

PressPass: How will this performance-based testing initiative impact IT organizations?

Valvano: IT professionals who achieve certification where performance-based testing is part of the exam have demonstrated real world competency and the ability to perform tasks required by a specific job role. This makes IT professionals better qualified to resolve problems faster and that translates into real cost savings and improved technical readiness. Performance-based testing hones and validates skills. The IT organization benefits because with highly skilled IT professionals it is better positioned to provide a higher level of service to its internal and external customers.

Trask: It’s all about instilling confidence in IT managers that Microsoft certification is a true and meaningful indicator of someone’s ability to perform a job. Whether used as a barometer for smart hiring, or with existing staff to increase the likelihood of having successful IT projects, like an Active Directory deployment, adding the simulations will give IT managers the added assurance that Microsoft certification can be used as a factor in making decisions.

PressPass: What’s it like to take one of the exams with simulations?

Valvano: It’s a “wow” experience. I’ve taken a lot of exams throughout my life – from school, into college, and then in my professional career -and this is a very different thing. Simulations incorporate many domains of skills in a single question. You’re in a simulated environment and the simulation records all the steps you take to solve the problem. Your score is based on the path you take and the outcome you achieve. It’s directly connected to what you do on a day-to-day basis in IT. What’s really being tested is: Do I know how to do this?

Trask: Microsoft is to be applauded for including real-world simulations within their certification exams. Demonstrating that you can complete tasks offers a level of assurance we believe IT managers will welcome with open arms.