18 fascinating developer war stories and postmortems

Project postmortems or retrospectives can teach us much about software development, so it's odd that more teams don't do them—if not for the benefit of future developers, then at least for the benefit of their own teams. 

The terms used to describe this follow-up narrative vary based on the development or management philosophy to which the team ascribes. One of these is the “Santayana Review,” named for the philosopher George Santayana, who famously said, “Those who cannot remember the past are condemned to repeat it.” In agile, you’ll hear the term "retrospective" used. In military contexts they call it an "after action review." Kanban? "Improve collaboratively." Lean software development? "Amplify learning." And there are others.

Hidden within the reflection process—whatever you choose to call it—are all sorts of opportunities to learn, improve, and maybe even be a little bit entertained. In this list we've collected an assortment of provocative and interesting war stories from a few software development blogs and a lot of books.

Testing in the Agile Era: Top Tools and Processes

Postmortem blogs

Let’s start by looking at some of the online resources available, which include real project stories and postmortem opinions from which we can learn.

Folklore.org

Folklore.org has captured 121 stories about the development of Apple's original Macintosh computer from the team members who built it. Most of its stories are short anecdotes, and often there are several different perspectives available for many of the interesting events that took place. 

Behind the Games, "The Final Hours"

Coding Horror's Jeff Atwood recommends Geoff Keighley's Behind the Games series, which “while not quite postmortems,” are written in that spirit. He goes behind the scenes at major gaming companies such as id and Blizzard Entertainment. He also dissects software development projects in the game industry such as Mass Effect 3, Tomb Raider, and Titanfall in a series called “The Final Hours.”

Code of Honor

Patrick Wyatt is a game developer with an amazing history. Now a senior principal engineer at Amazon Game Studios, he worked on landmark PC games including Warcraft I-III, World of Warcraft, Starcraft, Diablo I and II, and Guild Wars. He writes about his work on those games in his blog, Code of Honor. Although it has not been updated since Halloween 2013, it's well worth a read if you haven't already found it. Wyatt also mentions Stay Awhile and Listen: How Two Blizzards Unleashed Diablo and Forged a Video-Game Empire, a series of books written by David L. Craddock about the making of the games World of WarCraft and Diablo/Diablo II for which Wyatt was interviewed.

Engineering blogs

While they probably won't divulge anything juicy or highly negative about their workplace, these are some of the best blogs for finding everyday stories about how some of the most high-profile tech companies are operating. Any omissions are probably due to the fact that some engineering blogs don't ever post stories about their own internal journeys.

Facebook: Facebook Engineering

Twitter: Twitter Engineering

Linkedin: Blog | LinkedIn Engineering 

Dropbox: Dropbox Tech Blog

Uber: Uber Engineering Blog

Spotify: Spotify Labs

Instagram: Instagram Engineering (no longer posting, but there are some good ones in the archive)

Buffer: Overflow Buffer

Netflix: The Netflix Tech Blog

Airbnb: Airbnb Engineering

Flickr: code.flickr.com

Etsy: Code as Craft

Yelp: Yelp Product & Engineering Blog 

Medium: Medium Engineering

Basecamp: Signal v. Noise

Slideshare: Slideshare blog on LinkedIn Engineering

Soundcloud: SoundCloud Backstage

Salesforce: Salesforce Engineering

Gaming the system

A few more online resources that focus on game development postmortems can be found at the Gamasutra and The Pixel Prospector websites, as well as in the Adobe Developer Connection. Not only are there excellent game development postmortems online, as I mentioned above, there are also some excellent books on the subject. 

Masters of Doom: How Two Guys Created an Empire and Transformed Pop Culture

by David Kushner

How did John D. Carmack, John Romero, and id Software create the cult classic game Doom? This book has all the details, starting with the development of Commander Keen.

The Making of Karateka and The Making of Prince of Persia

by Jordan Mechner

As a young game designer, Jordan Mechner kept detailed journals describing his work coding two computer games, one of which would become the high-profile video game The Prince of Persia.

Postmortems from a Game Developer:
Insights from the Developers of Unreal Tournament, Black and White, Age of Empires, and Other Top-Selling Games

by Austin Grossman

The title tells it all. This book collects postmortem tales from games ranging from Age of Empires to X-Wing. Here's the top review on Amazon:

"Each postmortem is full of advice about designing, producing, and shipping a video game—advice from the developers themselves—allowing YOU to learn from THEIR mistakes. Technical gaffes, hiring snafus, political missteps, project management mistakes; anything that can go wrong in a software project HAS gone wrong in one of these stories." —S. Maher

War stories from the development trenches

Although it seems game development has its share of stories to tell, other software developers in various aspects of the industry have chimed in with their own war stories.

The Mythical Man-Month: Essays on Software Engineering

By Frederick P. Brooks Jr.

One book you should have on your shelf is this collection of essays first assembled in 1975 by “the father of the IBM System/360.” The Mythical Man-Month is filled with provocative information on the software creation process from start to finish. It is “as relevant today as when it was first published,” contends Christopher Byrnes, a Massachusetts-based software engineer.

Postmortems are not mentioned by name, but an entire chapter is devoted to determining “Why Did the Tower of Babel Fail?” in which Brooks discusses management audits.

The best quote from the book: "The bearing of a child takes nine months, no matter how many women are assigned."

Coders at Work

by Peter Seibel

Postmortems work because we learn from the triumphs and failures of others. There are tales from 15 different exceptional practitioners in Coders at Work. These include Donald Knuth, Brendan Eich (JavaScript, Mozilla), and Fran Allen.

Software Runaways

by Robert L. Glass

This 1998 book chronicles 16 different events gone wrong and the six factors contributing to those failures. Discussed are the aftermath of several high-profile software development project failures, including the United States Internal Revenue Service IT modernization project, the Denver International Airport’s baggage claim system, and the FAA's air-traffic control system. The book collects case studies previously published elsewhere, and there is value in having that information in one place.

Some of these stories are still relevant today, as this month's Delta outage proves.

Death March

by Edward Yourdon

In his book on epic IT project failures, Yourdon presents a more skeptical take on the postmortem value. Essentially, he says the benefit is debatable, since "many death march project failures are swept under the rug." Additionally, he notes, "The survivors who participated in the latter stages of the project are typically too burned out and exhausted to write the kind of memoirs we've come to expect from generals and former presidents. And even if they did, who reads these documents?"

Yourdon explains the factors contributing to these endless, doomed projects and advocates for simple incremental postmortems through the project to actually help those working on the project now. He calls these "mini-postmortems." The value of these is greater than attempting to record some halfhearted thoughts once a project has gone south.

Big projects

Several books are devoted to dissecting the development of a single piece of software. Among them:

Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software

by Scott Rosenberg  

This book describes the development process in full for the 2008 open-source calendar application created by Mitch Kapor and the Open Source Applications Foundation. At one point, Rosenberg notes, the process is not about building a program, but actually building Kapor’s organization, which had a roughly six-year lifespan. Some argue it helped fledge the Mozilla Foundation, of which Kapor was the founding chairperson.

Sweating Bullets: Notes about Inventing PowerPoint

by Robert Gaskins

As the head of the Microsoft group responsible for designing PowerPoint, Gaskins has some unique insights into the process of birthing a world-changing software project. Sweating Bullets tells that story. Bonus: Gaskins offers the PDF—all 516 pages—at his website.

Showstopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft

by G. Pascal Zachary

Showstopper! is an insider look at the development of Windows NT. As such, it is one giant postmortem by a third party, in this case, a Wall Street Journal reporter who presents an unvarnished look into the process.

The Future Was Here: The Commodore Amiga

by Jimmy Maher

Along these same lines, there was an array of software developed for the Commodore Amiga, one of the first personal computers. In explaining the history of the platform, Maher takes a look at the Boing demo and the takeaways from its development.

FoxTales

by Kerry Nietz

Once deep within the trenches at Fox Software, Nietz describes the company’s trajectory from a family-owned startup to a massive success that would be sold to Microsoft in the early 1990s. The company was built on the fortunes of a dBase clone known as FoxBASE+ for the nascent personal computing market. Nietz also has a website, but he has eschewed writing code in favor of writing Amish zombie novels since leaving Microsoft.

Learning from First Responders: When Your Systems Have to Work

by Dylan Richard

In 20 pages, Richard explains how the Obama for America team managed to get 40 people focused on building a suite of tools in 18 months for the president’s re-election campaign. An entire chapter is aptly devoted to “Reflections.”  This short read is freely available from the publisher in exchange for your email address. Along those same lines, The Atlantic's "The Secret Startup That Saved the Worst Website in America: How a team of young people, living in a repurposed McMansion in Maryland, helped rebuild Healthcare.gov" unpacks the drama associated with the Healthcare.gov web portal. 

Stranger than fiction

Aversion to failure is why many organizations involved in software development shy away from the postmortem process. It’s deemed an admission of failure rather than viewed as an opportunity to learn and change. It’s also why many of these stories haven’t been told. Since organizations of all sizes are shy about making their mistakes public, some authors have chosen to take the approach of fictionalizing the software development process, start to finish.

The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

by Gene Kim, Kevin Behr, and George Spafford

As the name implies, this book presents a novelized approach to walking through an IT project. Poor Bill needs to get Parts Unlimited’s newest initiative up and running. If not—cue the scary music—his department will be outsourced!

Yes, it goes beyond the postmortem, but there is value in seeing the entire development process, even if presented under the guise of fiction.

The Human Side of Postmortems and Beyond Blame: Learning from Failure and Success

by Dave Zwieback

Two books by Zwieback address how to effectively learn from technology failures. While he delves into the value of the postmortem in one book, Beyond Blame addresses the value in moving forward to ensure better approaches to determining how complex systems fail. It's framed within a story format like The Phoenix Project.

Zwieback also has a rarely updated blog devoted to brief observations crossing all aspects of DevOps, including video of a 2013 conference presentation on injecting various human factors—stress, biases—into postmortems. He posts on Twitter more regularly and responds to a post about the use of the term “postmortem,” stating “It’s not just the name, but the practice. My hope is that using ‘Learning Review’ reminds folks of desired outcome.”

“The Learning Review” is described in detail in Beyond Blame and a framework for developing a comprehensive, realistic learning review is provided in the book. Zwieback infuses the framework essentials with his own reading suggestions, including Thinking, Fast and Slow, by Daniel Kahneman, and Drop the Pink Elephant: 15 Ways to Say What You Mean, by Bill McFarlan. Better yet, if you want to read The Human Side of Postmortems now, the Kindle edition is freely available. (Thank you, O’Reilly!)

You may also wish to go back to one of the articles that helped start this continual improvement movement across countless industries: “Building a Learning Organization,” by David A. Garvin. His article in the July–August 1993 Harvard Business Review explains the importance of learning to move forward effectively and productively, stating that “In the absence of learning, companies—and individuals—simply repeat old practices.”

The article is filled with examples from IBM, Boeing, Xerox, and other Fortune 50 firms. Yes, some of them may be shells of their former glory, but that’s beside the point. Garvin gives great information on the value of creating an environment for continual learning across organizations of all sizes.

Learn from learned scholars

Since we’re veering toward the academic, it’s an excellent time to point out that there is an endless amount of academic journals that provide project postmortems from any number of perspectives.

Two papers you may want to read are “Postmortem reviews: purpose and approaches in software engineering,” by Torgeir Dingsøyr, published in Information and Software Technology in 2005, and “Software engineering projects may fail before they are started: postmortem analysis of five cancelled projects,” published in the Journal of Systems and Software.

You may wish to search through ResearchGate, IEEE Xplore Digital Library, or Google Scholar for full-text editions of papers on the topic. In some cases, only abstracts are offered.

Make new code, keep some old; some is silver and some is gold

Although many people are reluctant to discuss their process, much less their failures, there is great value in plumbing these stories for information and insight. As  Mike Gunderloy points out in his article—The Project Postmortem: An Essential Tool for the Savvy Developer

Software postmortems, performed consistently, are a key part of bringing a development organization from chaos to smooth, repeatable functioning. In fact, if your development efforts are completely disorganized, postmortems can be a great way to start turning things around, because they will help you identify and keep the good parts while finding and throwing out the bad parts. If you're not already using this essential tool, it's never too late to start.

What are the books or blog posts you have found to be valuable and/or entertaining sources of postmortem stories? Please add your suggestions and ideas in the comment section!

Topics: Dev & TestDevOps