Practical Technology

If you were to list all the reasons why Obama beat Romney in the 2012 presidential race, chances are DevOps, the cloud, and open-source software (OSS) wouldn’t be on your list. They should be. As Harper Reed, the CTO of Obama for America explained in his recent Palmetto Open Source Conference (POSSCON) speech, all these technologies played a major role in the campaign. Or, as the New York Times explained after the election: “Technology doesn’t win political campaigns, but it certainly is a weapon — a force multiplier, in military terms.”

Whatever your own politics, and regardless of who you think ought to have won the 2012 presidential election, there’s no question that technology played a huge part in Obama’s success. Reed explained that the campaign, which amounting to “creating a billion-dollar enterprise from zero in 18 months,” couldn’t have happened without these open-source, Internet-centric technologies.

To do this, Reed and his team chose to use an open, cloud-based IT model. “This kept costs down and consistent, even at the most demanding times,” he said. Specifically, Reed chose Ubuntu as the server operating system, while generally running instances on Amazon Web Services (AWS).” Why Ubuntu? Because, he said, “It’s a cost-effective OS that’s stable and reliable and that scales easily as required.” Or, as Reed summed it up: “I love, love, Ubuntu.”

The campaign’s DevOps team created about 290 deployed products on top of this platform of anywhere from 100 to 1,000 Ubuntu AWS instances.

But the success wasn’t only in the choice of software tools. According to Reed, the fundamentals were, “Build a great team, practice failure, and use open source.”

Starting from the top, Reed believes that to put together a top team you must hire the best and the smartest people. “A’s hire A’s and B’s hire C’s,” Reed said. “Always hire people smarter than you.”

Nor should any project manager wait for a job requisition. You should always be closing the deal with potential employees, he advised. If you find the right person for the job, hire them as soon as possible and put them to work. If they don’t work out, bite the bullet. “Do not be afraid to fire people, because bad work habits can become infectious,” he added.

At the same time, Reed much preferred to hire OSS pros. His favorite hiring question was, “What’s in your GitHub?”

Once the campaign had the team in place, it had to produce code as fast as possible. Working programs were not enough. The team had to measure everything it could to make sure those programs were delivering the goods.

The only way you can find out the difference between functional and usable is to talk to your users, Reed said. That meant that it was vital to get the user experience for the campaign workers on the ground. He added that this was the first time any political campaign had hired a UX designer.

With the right team in place, and the code starting to ship, the next step was to practice failure. While “failing sucks,” he said, it’s better to fail early and move on than to fail when it really matters.

To make sure that failures weren’t campaign killers, Reed used Live Action Role-Playing (LARP) exercises. “We worked through every possible disaster situation,” he said. “We did three actual all-day sessions of destroying everything we had built.” Using scenarios Reed described as being like the movie Groundhog Day, the team role-played having entire systems fail, seeing how they failed, figuring out to work around the failure, and then reset everything. Then they would role-play how another mission critical application might fail, and repeat the process.

It worked. In the last critical weeks before the election, the campaign had no IT failures.

The same can’t be said for Romney’s campaign’s software stack, which totally failed on Election Day.

At the same time, as the Obama campaign went rolling up to the big day, there were some software failures: the usual just-before-production hiccups you get in any project. When that happened, the team never rolled-back software. Instead, they rolled forward and did another release. “Rolling back was safer, but it hurt users, so the developers kept pushing forward,” said Reed.

Despite all this focus on failures and what could go wrong, Reed said that the development team didn’t focus on its failures. “Instead, you build on them,” he explained. “We talked about our successes and we celebrated wins every Friday with a ‘retro.’ This was a look back at our successes for the week, no matter how big or small. It inspired the team to keep going and look forward to the getting back to work the next week.”

As you’ve gathered by now, the Obama for America technology team had to move very, very fast. They couldn’t have done it without OSS. “We stood on the shoulder of giants, and open source enabled us to stand taller,” said Reed.

The results speak for themselves. In less than two years, Reed and his crew created a powerful 21^stcentury IT stack that delivered the mission-critical goods in a presidential election – a ship date that cannot be slipped. All-in-all, their accomplishments speaks volumes from what you can do with smart use of the cloud, DevOps technology, and open-source software.

Open Source Software Helped Obama Win the 2012 Election Campaign. A version of this story first appeared on SmartBear.