Project postmortems or retrospectives can teach us much about software development, so it's odd that more teams don't do them—if not for the benefit of future developers, then at least for the benefit of their own teams.
The terms used to describe this follow-up narrative vary based on the development or management philosophy to which the team ascribes. One of these is the “Santayana Review,” named for the philosopher George Santayana, who famously said, “Those who cannot remember the past are condemned to repeat it.” In agile, you’ll hear the term "retrospective" used. In military contexts they call it an "after action review." Kanban? "Improve collaboratively." Lean software development? "Amplify learning." And there are others.
Hidden within the reflection process—whatever you choose to call it—are all sorts of opportunities to learn, improve, and maybe even be a little bit entertained. In this list we've collected an assortment of provocative and interesting war stories from a few software development blogs and a lot of books.
Postmortem blogs
Let’s start by looking at some of the online resources available, which include real project stories and postmortem opinions from which we can learn.
Folklore.org
Folklore.org has captured 121 stories about the development of Apple's original Macintosh computer from the team members who built it. Most of its stories are short anecdotes, and often there are several different perspectives available for many of the interesting events that took place.
Behind the Games, "The Final Hours"
Coding Horror's Jeff Atwood recommends Geoff Keighley's Behind the Games series, which “while not quite postmortems,” are written in that spirit. He goes behind the scenes at major gaming companies such as id and Blizzard Entertainment. He also dissects software development projects in the game industry such as Mass Effect 3, Tomb Raider, and Titanfall in a series called “The Final Hours.”
Code of Honor
Patrick Wyatt is a game developer with an amazing history. Now a senior principal engineer at Amazon Game Studios, he worked on landmark PC games including Warcraft I-III, World of Warcraft, Starcraft, Diablo I and II, and Guild Wars. He writes about his work on those games in his blog, Code of Honor. Although it has not been updated since Halloween 2013, it's well worth a read if you haven't already found it. Wyatt also mentions Stay Awhile and Listen: How Two Blizzards Unleashed Diablo and Forged a Video-Game Empire, a series of books written by David L. Craddock about the making of the games World of WarCraft and Diablo/Diablo II for which Wyatt was interviewed.
Engineering blogs
While they probably won't divulge anything juicy or highly negative about their workplace, these are some of the best blogs for finding everyday stories about how some of the most high-profile tech companies are operating. Any omissions are probably due to the fact that some engineering blogs don't ever post stories about their own internal journeys.
Facebook: Facebook Engineering
Twitter: Twitter Engineering
Linkedin: Blog | LinkedIn Engineering
Dropbox: Dropbox Tech Blog
Uber: Uber Engineering Blog
Spotify: Spotify Labs
Instagram: Instagram Engineering (no longer posting, but there are some good ones in the archive)
Buffer: Overflow Buffer
Netflix: The Netflix Tech Blog
Airbnb: Airbnb Engineering
Flickr: code.flickr.com
Etsy: Code as Craft
Yelp: Yelp Product & Engineering Blog
Medium: Medium Engineering
Basecamp: Signal v. Noise
Slideshare: Slideshare blog on LinkedIn Engineering
Soundcloud: SoundCloud Backstage
Salesforce: Salesforce Engineering
Gaming the system
A few more online resources that focus on game development postmortems can be found at the Gamasutra and The Pixel Prospector websites, as well as in the Adobe Developer Connection. Not only are there excellent game development postmortems online, as I mentioned above, there are also some excellent books on the subject.
Masters of Doom: How Two Guys Created an Empire and Transformed Pop Culture
by David Kushner
How did John D. Carmack, John Romero, and id Software create the cult classic game Doom? This book has all the details, starting with the development of Commander Keen.
The Making of Karateka and The Making of Prince of Persia
by Jordan Mechner
As a young game designer, Jordan Mechner kept detailed journals describing his work coding two computer games, one of which would become the high-profile video game The Prince of Persia.
Postmortems from a Game Developer:
Insights from the Developers of Unreal Tournament, Black and White, Age of Empires, and Other Top-Selling Games
by Austin Grossman
The title tells it all. This book collects postmortem tales from games ranging from Age of Empires to X-Wing. Here's the top review on Amazon:
"Each postmortem is full of advice about designing, producing, and shipping a video game—advice from the developers themselves—allowing YOU to learn from THEIR mistakes. Technical gaffes, hiring snafus, political missteps, project management mistakes; anything that can go wrong in a software project HAS gone wrong in one of these stories." —S. Maher
War stories from the development trenches
Although it seems game development has its share of stories to tell, other software developers in various aspects of the industry have chimed in with their own war stories.
The Mythical Man-Month: Essays on Software Engineering
By Frederick P. Brooks Jr.
One book you should have on your shelf is this collection of essays first assembled in 1975 by “the father of the IBM System/360.” The Mythical Man-Month is filled with provocative information on the software creation process from start to finish. It is “as relevant today as when it was first published,” contends Christopher Byrnes, a Massachusetts-based software engineer.
Postmortems are not mentioned by name, but an entire chapter is devoted to determining “Why Did the Tower of Babel Fail?” in which Brooks discusses management audits.
The best quote from the book: "The bearing of a child takes nine months, no matter how many women are assigned."
Coders at Work
by Peter Seibel
Postmortems work because we learn from the triumphs and failures of others. There are tales from 15 different exceptional practitioners in Coders at Work. These include Donald Knuth, Brendan Eich (JavaScript, Mozilla), and Fran Allen.
Software Runaways
by Robert L. Glass
This 1998 book chronicles 16 different events gone wrong and the six factors contributing to those failures. Discussed are the aftermath of several high-profile software development project failures, including the United States Internal Revenue Service IT modernization project, the Denver International Airport’s baggage claim system, and the FAA's air-traffic control system. The book collects case studies previously published elsewhere, and there is value in having that information in one place.
Some of these stories are still relevant today, as this month's Delta outage proves.
Death March
by Edward Yourdon
In his book on epic IT project failures, Yourdon presents a more skeptical take on the postmortem value. Essentially, he says the benefit is debatable, since "many death march project failures are swept under the rug." Additionally, he notes, "The survivors who participated in the latter stages of the project are typically too burned out and exhausted to write the kind of memoirs we've come to expect from generals and former presidents. And even if they did, who reads these documents?"
Yourdon explains the factors contributing to these endless, doomed projects and advocates for simple incremental postmortems through the project to actually help those working on the project now. He calls these "mini-postmortems." The value of these is greater than attempting to record some halfhearted thoughts once a project has gone south.
Big projects
Several books are devoted to dissecting the development of a single piece of software. Among them:
Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software
by Scott Rosenberg
This book describes the development process in full for the 2008 open-source calendar application created by Mitch Kapor and the Open Source Applications Foundation. At one point, Rosenberg notes, the process is not about building a program, but actually building Kapor’s organization, which had a roughly six-year lifespan. Some argue it helped fledge the Mozilla Foundation, of which Kapor was the founding chairperson.
Sweating Bullets: Notes about Inventing PowerPoint
by Robert Gaskins
As the head of the Microsoft group responsible for designing PowerPoint, Gaskins has some unique insights into the process of birthing a world-changing software project. Sweating Bullets tells that story. Bonus: Gaskins offers the PDF—all 516 pages—at his website.
Showstopper!: The Breakneck Race to Create Windows NT and the Next Generation at Microsoft
by G. Pascal Zachary
Showstopper! is an insider look at the development of Windows NT. As such, it is one giant postmortem by a third party, in this case, a Wall Street Journal reporter who presents an unvarnished look into the process.
The Future Was Here: The Commodore Amiga
by Jimmy Maher
Along these same lines, there was an array of software developed for the Commodore Amiga, one of the first personal computers. In explaining the history of the platform, Maher takes a look at the Boing demo and the takeaways from its development.
FoxTales
by Kerry Nietz
Once deep within the trenches at Fox Software, Nietz describes the company’s trajectory from a family-owned startup to a massive success that would be sold to Microsoft in the early 1990s. The company was built on the fortunes of a dBase clone known as FoxBASE+ for the nascent personal computing market. Nietz also has a website, but he has eschewed writing code in favor of writing Amish zombie novels since leaving Microsoft.
Learning from First Responders: When Your Systems Have to Work
by Dylan Richard
In 20 pages, Richard explains how the Obama for America team managed to get 40 people focused on building a suite of tools in 18 months for the president’s re-election campaign. An entire chapter is aptly devoted to “Reflections.” This short read is freely available from the publisher in exchange for your email address. Along those same lines, The Atlantic's "The Secret Startup That Saved the Worst Website in America: How a team of young people, living in a repurposed McMansion in Maryland, helped rebuild Healthcare.gov" unpacks the drama associated with the Healthcare.gov web portal.
Stranger than fiction
Aversion to failure is why many organizations involved in software development shy away from the postmortem process. It’s deemed an admission of failure rather than viewed as an opportunity to learn and change. It’s also why many of these stories haven’t been told. Since organizations of all sizes are shy about making their mistakes public, some authors have chosen to take the approach of fictionalizing the software development process, start to finish.
The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win
by Gene Kim, Kevin Behr, and George Spafford
As the name implies, this book presents a novelized approach to walking through an IT project. Poor Bill needs to get Parts Unlimited’s newest initiative up and running. If not—cue the scary music—his department will be outsourced!
Yes, it goes beyond the postmortem, but there is value in seeing the entire development process, even if presented under the guise of fiction.
The Human Side of Postmortems and Beyond Blame: Learning from Failure and Success
by Dave Zwieback
Two books by Zwieback address how to effectively learn from technology failures. While he delves into the value of the postmortem in one book, Beyond Blame addresses the value in moving forward to ensure better approaches to determining how complex systems fail. It's framed within a story format like The Phoenix Project.
Zwieback also has a rarely updated blog devoted to brief observations crossing all aspects of DevOps, including video of a 2013 conference presentation on injecting various human factors—stress, biases—into postmortems. He posts on Twitter more regularly and responds to a post about the use of the term “postmortem,” stating “It’s not just the name, but the practice. My hope is that using ‘Learning Review’ reminds folks of desired outcome.”
“The Learning Review” is described in detail in Beyond Blame and a framework for developing a comprehensive, realistic learning review is provided in the book. Zwieback infuses the framework essentials with his own reading suggestions, including Thinking, Fast and Slow, by Daniel Kahneman, and Drop the Pink Elephant: 15 Ways to Say What You Mean, by Bill McFarlan. Better yet, if you want to read The Human Side of Postmortems now, the Kindle edition is freely available. (Thank you, O’Reilly!)
You may also wish to go back to one of the articles that helped start this continual improvement movement across countless industries: “Building a Learning Organization,” by David A. Garvin. His article in the July–August 1993 Harvard Business Review explains the importance of learning to move forward effectively and productively, stating that “In the absence of learning, companies—and individuals—simply repeat old practices.”
The article is filled with examples from IBM, Boeing, Xerox, and other Fortune 50 firms. Yes, some of them may be shells of their former glory, but that’s beside the point. Garvin gives great information on the value of creating an environment for continual learning across organizations of all sizes.
Learn from learned scholars
Since we’re veering toward the academic, it’s an excellent time to point out that there is an endless amount of academic journals that provide project postmortems from any number of perspectives.
Two papers you may want to read are “Postmortem reviews: purpose and approaches in software engineering,” by Torgeir Dingsøyr, published in Information and Software Technology in 2005, and “Software engineering projects may fail before they are started: postmortem analysis of five cancelled projects,” published in the Journal of Systems and Software.
You may wish to search through ResearchGate, IEEE Xplore Digital Library, or Google Scholar for full-text editions of papers on the topic. In some cases, only abstracts are offered.
Make new code, keep some old; some is silver and some is gold
Although many people are reluctant to discuss their process, much less their failures, there is great value in plumbing these stories for information and insight. As Mike Gunderloy points out in his article—The Project Postmortem: An Essential Tool for the Savvy Developer
Software postmortems, performed consistently, are a key part of bringing a development organization from chaos to smooth, repeatable functioning. In fact, if your development efforts are completely disorganized, postmortems can be a great way to start turning things around, because they will help you identify and keep the good parts while finding and throwing out the bad parts. If you're not already using this essential tool, it's never too late to start.
What are the books or blog posts you have found to be valuable and/or entertaining sources of postmortem stories? Please add your suggestions and ideas in the comment section!
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
