Meet the ‘Plumbers’ Powering ‘Halo 4’ Infinity Multiplayer

REDMOND, Wash.

Oct. 31, 2012
Meet Jerry Hook and Tamir Melamed, the hardest-working plumbers in the video game world not named Mario or Luigi.

Hook and Melamed lead the engineering team laying the subterranean IT structures that will power every pixel of the multiplayer experience in “Halo 4.” Everything fans experience online – stats, screenshots, the simple joy of blasting a friend or stranger to smithereens – depends on the infrastructure they’ve built over the past year and half. That infrastructure is supported by Windows Azure, which provides the team with the affordable scalability they need to keep a game like “Halo 4” running smoothly for fans.

The average gamer probably doesn’t think about the virtual machines, cloud services and other technologies that power their fun – at least not as long as everything runs smoothly. When “Halo 4” launches Nov. 6, millions of fans around the world will simultaneously push the game’s back-end scaffolding to the limit. Any glitch, any hang-up, and the engineering team gets thrust into the spotlight.

“That’s the lot of the plumbers of the world,” says Melamed, principal development manager for Microsoft’s 343 Industries. “You only think about them when something goes wrong.”

If Hook and Melamed are concerned about the launch, they don’t show it. They’re both naturally modest and laid back – probably by necessity. When the first “Halo” game launched more than a decade ago, the video game industry’s rhythm was more or less straightforward: create a bunch of discs, ship them, and start work on the next release.

Tamir Melamed and Jerry Hook Tamir Melamed and Jerry Hook

October 30, 2012
Tamir Melamed (L), principal development manager for 343 Industries and Jerry Hook, executive producer of 343 Industries’ Section 3 team.

Things have changed. Since “Halo 2,” the franchise has supported online multiplayer games via Xbox LIVE. Today the Halo Waypoint site is open 24/7, serving as a central hub for fans craving online games and Halo-related content. The upcoming “Halo” release features Infinity Multiplayer, a vastly expanded suite of multiplayer modes, weapons, vehicles, armor abilities and more. And “Halo 4” takes the multiplayer experience even further with Spartan Ops, a series of episodic videos coupled with missions that will stream to users every week.

Hook, Melamed and their colleagues on 343 Industries’ Section 3 team will be responsible for keeping things running for months, even years after the official launch. (The team takes its name from the Halo universe; fittingly, Section 3 refers to a military branch responsible for black ops.)

“‘Halo’ from here on out does not end,” says Hook, executive producer at Section 3. “It’s a sustained franchise. Players will always feel engaged.”

To keep things up 24/7, they decided to take the franchise’s existing infrastructure – built over a decade by the Bungie game studio – and move it to Windows Azure. The team says Azure gives them a robust, cost-effective way to scale up and scale down as demand ebbs and flows across the globe.

Meanwhile, they’ve built up a team of engineers that possesses serious coding chops and the ability to stay sane in the always-on world of live services.

“We know we can’t anticipate the 101 things that will go wrong,” Hook says. “The only thing we can control is ensuring our team responds appropriately to those situations.”

Hook and Melamed recently sat down with the Microsoft News Center to chat about “Halo 4,” the gaming industry’s shift to a services-oriented world, and how to stay cool when the building’s on fire.

MNC: So how does your behind-the-scenes work show up in “Halo 4?

Melamed
: There’s a whole new slew of scenarios we enable on the back-end, either in the game or on Waypoint, that people enjoy: screenshots, their stats and the ability to compare to friends, different game modes. We make sure if someone is hassling someone else in the game you can ban them. All those abilities really help users enjoy the game more.

Hook: From my perspective, part of what 343 Industries is trying to do is have a conversation with fans. When you talk about service-driven game development, when you talk about new modes like Spartan Ops, these start becoming more long-tail models for entertainment instead of release a disc, go dark and come back. That means your community engagement maintains through one title to the next title as much as possible.

So there’s what Tamir was talking about in the game, like screenshots and map packs, but you also see more entertainment like Spartan Ops. Think about a TV show: you don’t just launch and then go dark until the next year. With Spartan Ops, we’ll release an episode every week. Each episode is comprised of five missions with stories similar to your campaign. So for the first time when you purchase a disc, you don’t just have one campaign but you get two: the standard one and an ongoing, episodic one.

Commander Sara Palmer, Spartan Ops, UNSC Infinity Commander Sara Palmer, Spartan Ops, UNSC Infinity

October 30, 2012
“Halo 4” takes the multiplayer experience even further with Spartan Ops, a series of episodic videos coupled with missions that will stream to users every week.

Melamed
: Fans will be able to come back every week and experience a new episode. They can jump in at any time, or watch and play old episodes – as you would similar to DVR. Players will be excited to come back to the title for the new experiences.

MNC: Just how big is this shift to a services world?

Hook: It’s as significant as when Xbox LIVE brought multiplayer to the living room. You’re changing the entire paradigm of gaming. It’s been kind of gradual, but I think we’ve hit the start of the S curve. And you’ll start to see it shifting across most of the major publishers.

Melamed
: And it’s not just a matter of ‘you need to add services.’ The most important thing is what new experiences you can create and what new ways can the users can interact and have fun. How do you tap powerful services so the game is even more enjoyable than it is now, more social, more related to your friends?

You have to start thinking about your game as a service. In a more rapid cycle of delivering content, you can see what fans like, what they don’t like. You can give the creative guys the ability to try things so they don’t have to lock on one scenario and do that for the next three years.

Hook: That’s the goal. When Tamir talks about social, we’re thinking about how to bring that into the entertainment experience. How can you enable fans to have an impact into what you’re doing? Think of the power that is available not just for the hardcore fans but also for the casual fans: When you step into a new world, or a new universe like the “Halo,” you can start from anywhere.

So say you like sci-fi and read a book like “Halo: The Fall of Reach.” How do you get those people from reading a book to playing one of the entertainment experiences? Not everyone will necessarily like a first-person shooter, so what other experiences can we start adding to bring those more casual users into our universe?

Melamed
: That opens new abilities for us to build more interesting things in the game online.

MNC: Why did you decide to build your infrastructure on Windows Azure?

Melamed
: The scale we need to support on Halo 4 is fairly high. We’re in the hundreds of thousands of requests per second, and we could not have done this without Azure. Well, maybe we could, but it would have been an astronomical cost.

For us, Azure is critical. We don’t know what the demand will be. How do we get our services ready for x million or y million? If you have your own data center, you have to aim for the sky, then buy hardware for the sky and hope the fans will come. And if your sky based on your budget isn’t high enough, you basically take the system down. With Azure, you can say, ‘I need another 1,000 servers’ and your fans keep playing.

343 Industries Engineering Team 343 Industries Engineering Team

October 30, 2012
Members of 343 Industries’ engineering team is responsible for building the infrastructure that powers all the virtual machines, cloud services and other technologies in “Halo 4.”

Hook: And then there’s the converse of that. Most games have a peak in the first three months from a concurrency perspective, and after that, you’ll start reducing to a more sustained level. We hope to break that model with Spartan Ops. We think we’ll be able to have that longer sustained. So with Azure you have both sides of fence: you can expand and reduce as you need to.

MNC: How do you prep your team for launch, when they start running a 24/7 service?

Melamed
: This team has to always be on. We can’t be in a situation where we peak and then go to Vegas for a couple months to relax.

Hook: We’re a lot more even-keeled. The games team will burn really, really hot over a period of time. We have to balance ourselves where we don’t have those peaks and burns because we know the team will have to support the game well after launch.

Melamed
: We’ve run plenty of tests, but we know things will be different at launch. When you have people in Singapore playing against someone in Argentina, you have different network configurations, different firewalls, all kinds of issues, and scenarios you can’t simulate. It’s incredibly complex. You need to have your engineers ready to support and understand and analyze problems that come up.

We have a plan for the first month after launch about how we’ll have people work in shifts, how we’ll make sure we have people around all the time while making sure they have enough time to relax and not burn out. We actually have a war room where we’ll coordinate with the 15 or 20 teams across Microsoft helping us.

MNC: Will Master Chief
be
in the war room?

Melamed
: There actually will be a designated Master Chief running the war room. We’ll rotate different people in. I’m just a developer though. The program managers are the Master Chiefs. I’m just a grunt.

Hook: Please note Jerry rolled his eyes and shook his head at that comment.

MNC: So how do you two stay cool?

Hook: Me, I just resort to humor. Tamir is like this Plymouth Rock of even keel. I swear, the building could be on fire, and Tamir would turn to you and say ‘You know, we probably could use a bit more water over there. But don’t worry; just relax.’ That’s Tamir. He’s Zen.

Melamed
: It’s just being old and less responsive. That’s all.

MNC: What are your hobbies?

Hook: Sleep.

Melamed
: I try to see my kids from time to time, besides a few pictures here and there. The whole team is sacrificing for the title, no question. The level of engagement and dedication is pretty amazing.

MNC: If you’re working all the time, I guess that makes it easier to deal with the long, gray Seattle winter.

Melamed
: But we had a lovely summer. We walked between buildings sometimes for the vitamin D.

Hook: Wait, wait. We had summer?