New hire or new bot? He built an AI bot to improve mundane tasks for this team. Would they notice it wasn’t human?
Why one Microsoft employee built an AI bot for the team supporting Xbox game creators
It was your average “nice to meet you moment” between two employees.
Torey Allen met Rani Sobeck over email early last spring. Rani was a new hire on the team who worked remotely, and Allen figured she’d meet her in real life eventually.
But when a few weeks passed and she never saw Rani come into the Redmond office, she got curious. What did she look like? Where was she working from?
That’s when Allen decided to sniff around.
“I looked her up online and saw all of these Halo references,” said Allen, who works as an associate producer for Microsoft’s game lab and birthplace of the game Halo, 343 Industries. Allen verified that the name Rani Sobeck is part of Halo lore, a character whose brain was used as the basis for artificial intelligence (AI). Then Allen was suspicious: “This person is either really in love with Halo and changed her name legally, or there’s something weird going on here.”
To find out, Allen asked Dan Price, Rani’s manger. That’s when Price let her in on his little secret: Rani Sobeck was an AI bot. He’d created Rani to help the team see and solve problems faster.
Price’s team had been seeing too much repetition in their daily tasks and he wanted to free them up to work on more interesting, complex problems. But could he build his solution and then keep it secret from the team long enough to test and perfect its functionality?
Naturally artificial intelligence
Price’s fascination with AI wasn’t new, but it came to a head a few weeks prior to his conversation with Allen. Price had just finished binge-watching the last season of “Person of Interest.” One of the characters on the television show, an AI bot named the Machine, possessed critical information that could save people’s lives, if she could be given the freedom to do so. Price was fascinated with the natural interaction between AI and humans.
In the show, the Machine easily and naturally integrates into everyday human life.
“If you think about classic ‘Star Trek’ where they are always saying ‘computer, locate this person or do this thing,’ that’s a very disconnected experience,” said Price.
In this instance, the Machine is an all-knowing, all-seeing super computer that is a natural part of people’s daily lives. He felt inspired and intrigued . . . could he create something like the Machine? A software engineer who had experience with AI courses during his Master’s degree education, Price knew he had enough background to at least tinker with it.
These theoretical thoughts were bouncing around in Price’s head at the same time that he and his colleague Tom Hill were contending with some less-than-theoretical problems at work.
Price’s IT team supports the more than 1,500 machines that Microsoft employee game creators, use; Hill’s team supports the software that is running on those machines. Their biggest task is to build and maintain tools that help people who create games do their jobs.
But recently, Price and Hill had become increasingly fed up with their lack of tools to manage support tickets.
“We get all of these tickets that come to our desktop queue from all of these users,” said Price.
But many of the tickets are duplicates, repetitive, or easy one-off fixes that no has time to log. Trouble tickets are not just a work tool for assigning jobs—they are essential for product improvement because they track issues, revealing patterns that could indicate the need for a more systemic response. But without a consistent record, there’s no way to track recurring issues or measure performance against any kind of metrics. And when someone fixes something ad hoc and leaves no data trail, bigger problems arise—problems that can threaten the business.
Hill likened the support ticket inefficiencies to getting “100 little cuts throughout the day.”
The team would be moving along through the queue of issues, resolving problems quickly, but then be too busy to close tickets for minor issues. Plus, there was no auto-sort or assigning the level of urgency or to the right team. They would all just land in the same place, creating a homogenous pool of tickets where anyone on the team would have to grab and guess.
“It became a task we couldn’t really name or quantify. Those tasks, when they stand on their own, don’t seem that big,” Hill said. “But if you look at the quantity of them—we just decided we don’t want to be cut this way anymore.”
Price asked himself if he could simply figure out what type of issue they were having at a high level and then make sure a ticket got opened with the right team and the right priority. It wasn’t a hard task, just a tedious one.
He wished he’d had his own machine like on the television show. He decided to try his hand at combining his love of AI with his passion for tools that make work more efficient; the marriage produced his first bot: Rani Sobeck.
A secret bot-dentity
Price knew he’d have to keep it simple to start. So he focused on making an AI that could open and close support tickets and maybe learn to do more later. He also wanted to push himself; could he give it a personality? Could he make the interaction feel natural and human-like?
“I wanted my team to feel like the AI bot was part of the team, like the interaction would be the same as talking to Jane who sits next to you,” he said.
He’d have to perform his own Turing test to launch the bot—and tell no one. It would be the only way to see how well the bot would mesh with humans.
Rani would interact with team members solely through email. So Dan built the initial framework with custom tools like PowerShell. Then, to help train Rani, he reviewed hundreds of emails sent to the support email to see if he could find a pattern. He created buckets: operating system issues, software issues, hardware issues, etc. It turned out that almost all of the issues that ended up coming to the team fit nicely into one of about 15 buckets.
He then created an initial set of keywords, like tags, for each of those areas and fed them to the Rani bot. But before he set her off to interact with coworkers, he trained Rani to forward the support request to him with Rani’s recommended suggestion for what she thought the issue was and assign a priority to the case. Every time Rani succeeded or failed at ticket sorting, she used that data to learn quickly. With each ticket, Rani was becoming more sophisticated and accurate.
The last step before deploying Rani was to write logic that would allow her to ask questions if she was unsure what to do with an issue that didn’t fit neatly into the 15 categories.
Then Rani’s deployment day arrived quietly. Price put her to work, but didn’t tell anyone. Over the next few months, Rani grew better and better at her job – learning from the tickets coming into the system.
After a few months, Price sent out a team email to welcome people who recently started in IT. When he introduced Rani, he made no mention of her secret bot-dentity. That’s when Allen, the team member who really wanted to meet Rani, started noticing replies to her email requests for IT help being answered by Rani.
Allen and other people on the team, including Tina Summerford, came to Price, saying how much they appreciated Rani, how “on it” she was, and how quickly she responded to requests. Surely Rani was working overtime, thought Summerford. She repeatedly praised Rani to Price and wanted to meet her.
That’s when Price decided to reveal Rani’s true nature. “I mean, at some point I actually felt bad,” he said.
Ironically, Price’s initial worry was that Rani wouldn’t be that helpful to the team. She had reactions that at first he hadn’t anticipated. For instance, “a user would send a ticket and I’d say, ‘Hey Rani, would you please go open a ticket for this’ and while I am doing that, someone else from my team is also asking her to do the same thing, for the same issue.”
Rani would do as told and create a ticket—twice. Price has since fixed that, and now she has learned not to repeat tasks. In fact, she responds with “No, Dan. You already asked me to do that one. I come to serve.”
“Oh yeah, she’s pretty sarcastic,” said Price, laughing. The other day, in a request to close a ticket, she responded to Dan, “Dan! You met your dreams and aspirations. Now take a bow – congratulations. Changed as requested.” She’s also capable of recognizing gratitude and teasing and even uses nicknames for the team (she calls Hill “mate” all the time, slang in Hill’s home country of the United Kingdom, which he thinks is pretty hilarious).
“It’s fun, but she’s also getting smarter,” Price said.
Releasing humans; not replacing them
Having Rani for the past year has completely changed Price’s daily workflow and provided insight into some consistent problems the team had never been able to identify before.
Now, he asks himself (and encourages his team to do the same) if a task is something he should be teaching Rani. If there is something he does over and over again every day, he subjects that task to a little calculation: add the daily time spent on that task (say, 30 minutes), then add up the hours it would take to teach the task to Rani (say, four hours). Four hours one time versus two and a half hours per week—the result is often conclusive: just teach Rani.
As for Price, he’s experiencing the high that comes from a passion and his work coming together. And his coworker Hill asked for his own AI bot after a few months, named Gary. In fact, there are seven AI bots now deployed on the team.
Hill said that the efficiencies they’ve gained have changed the culture on his team; there’s a lot less stress because people don’t have to spend tedious hours on mundane tasks or have to work at inconvenient times. For example, Gary now monitors the night shift.
For Price, the whole experiment has proven the value of using a bot to augment what people already do and free them up for more complex, sophisticated tasks.
“My staffing budget hasn’t changed; we’ve just allocated funds differently,” said Price.
“We haven’t replaced people by gaining efficiencies. We channel those efficiencies into humans doing other, more productive things.”
