Project management: A surefire way to kill your software product

For many years, I have observed that the quality of software produced by organizations is a decreasing function of their emphasis on project management over business value creation—that is, their obsession with predictability and efficiency over learning and adapting. Software development is fundamentally an exercise in learning, while the traditional command-and-control style of project management seeks to optimize the execution of known, repeatable processes at scale.

For software development, no significant developer activity is predictable or repetitive; if it were, the developers would have automated it already. In addition, learning is essentially a nonlinear process; it involves trying things that don’t work in order to discover what does work.

You might see linear progress for a while, but you don’t know what you don’t know, so there will be apparent setbacks. It is from these setbacks that one learns the truth about the system—what is really needed to make it work, to make it usable, and to make a difference for the users and the business.

In other words, the dirty little secret of software development is that projects don’t really exist. And they’re killing our products, teams, and software.

Projects don’t exist?

Projects, with respect to software development, are imaginary boxes drawn around scope and time in an attempt to “manage” things. This tendency is understandable, given the long fascination with so-called scientific management (a.k.a. Taylorism, a.k.a. Theory X), but these imaginary boxes do not reduce underlying complexity. On the contrary, they add unnecessary complexity and friction and invite a counterproductive temptation to focus on the box instead of the problem or product. This misplaced emphasis leads to some harmful delusions:

• Conformance to schedule is the same thing as success.
• Estimation accuracy is possible and desirable enough to measure and optimize for.
• The plan is perfect and guarantees success.
• The cost of forming and dissolving teams is zero.
• The cost of functional silo hand-offs is zero.
• The bigger and more comprehensive the plan, the better.
• Predictability and efficiency are paramount.

The last delusion is the heart of the problem: Traditional project-management attempts to optimize for predictability, for “efficiency.” Here's why. 

What’s wrong with predictable efficiency?

Isn’t predictability a good thing? Isn’t being efficient desirable?

Yes and no.

Yes, if achieving the a priori schedule is the critical challenge.

No, if you actually want to learn how to solve your problem, please the users, and add value to the business.

Predictability kills learning and innovation

Innovation and learning are incompatible with predictability. If you’re optimizing for predictability, you’re motivating people to hide the truth. You may have encountered projects where everything was fine, fine, fine until suddenly it wasn’t—and no one believed that it “suddenly” went bad; the truth was just masked for a while.

Even in “successful” projects, it’s entirely possible to deliver everything on time, under budget, with exactly the features specified, and still fail because the things that were delivered were not what was needed or wanted by the users.

This happens because project management emphasizes things that do not matter—estimates, time, conformance, ceremonies, and all of that—while ignoring the things that do matter: value generation, user satisfaction, group learning, team dynamics, root causes, information gaps, business insights, and innovation.

Regression to the mean is inevitable

Keep this up, and your high-performing team members—the ones who thrive on challenges, who like to innovate, who are capable of producing the breakthroughs and new ideas that provide strategic advantage to your business—will leave. Regression to the mean is practically guaranteed when plan conformance is the goal.

No risk, no experiments, and no learning; if you really have software that can be produced under these circumstances, chances are that software either is not worth writing in the first place or is a commodity you can just buy instead of build. If you’re trying to write substantially innovative software or to learn something deep and meaningful about your business or users, you cannot learn it by focusing on predictability.

Project management is not all bad, is it?

Many project managers (PMs) provide valuable assistance to their teams: clearing obstacles, looking ahead, taking care of the team’s morale, upskilling team members, and so forth. Note that none of these valuable activities involves tracking conformance to a plan or estimation accuracy.

As a project manager, if you find yourself encouraging the team to work overtime so that it hits its total story-point goal for a sprint, then you’ve become part of the problem. The notion of story-point velocity as a goal is a profound misunderstanding of agile.

What about planning?

A roadmap of where you intend to go is fine; it can be adjusted as you learn. An uppercase Project Plan, however, tends to take on the mantle of truth, the measure of success, at the expense of the product.

Process improvement is everyone’s responsibility—that’s why agile teams have retros—but in a project-mindset world, the process is already perfect, so improving it not only is no one’s responsibility but even suggesting that it might need improvement invites charges of heresy.

What do “good” PMs do?

So what do “good” project managers do? They clear the road ahead. They take care of their teams. They make sure the business vision is shared and understood across the board. They don’t micromanage, they don’t back-seat drive, and they definitely don’t ask, “Are we there yet?” every three minutes like a bored toddler on a long car trip. (Yes, “How’s it going?” is the same as, “Are we there yet?” so please, please stop doing that.) If you have a specific question, ask it; don’t interrupt the developers for a status report; you’re just making things take longer.

Aren’t deadlines important?

Short answer: no. Deadlines are usually arbitrary, based on wishful thinking, uninformed guesses, irrelevant external factors, and a giant bag of assumptions.

I know, I know: The customers need the widget upgrade by next Tuesday, and marketing wants to announce the new features at the big industry conference next month, and if we don’t hit the pre-holiday release date we will miss huge opportunities for sales, etc.

So what?

So, the team has to deliver it by the deadline, obviously.

This kind of statement is drowning in an unsavory gravy of assumptions, the most obvious being the assumption that it is possible to deliver quality and completeness by the deadline. You don’t know that, and you can’t know that; the development team doesn’t even know that, though it may express high confidence. But is that because the team members are truly confident, or because they have been trained to pretend to support the Plan no matter what?

Projects degrade architecture and quality

Focusing on projects can also compromise architectural integrity. For example, short-lived project teams judged on conformance to the schedule have ample incentive to take shortcuts and avoid refactoring. Furthermore, they are insulated from the longer-term pain of their decisions, so technical debt grows.

Unnecessary team changes and hand-offs further degrade the ability of the team to learn what is necessary to deliver what users want. If you find that your teams consistently deliver less and less as the code base grows, blame projects. In these situations, no amount of architectural principles, guidelines, or tools can prevent the eventual decay of software quality.

Product thinking over project thinking

Should we just throw out project management completely?

Yes, thank you, that would be great—and please take Scrum with you!

In an agile team using Kanban, for example, there is no “project” at all—there’s only a continuous stream of value-delivering work, prioritized by someone who keeps a finger on the pulse of the customer and validated with actual customers.

Zero time is wasted on elaborate long-term planning, Gantt charts, work breakdown structures, and so forth. Not that those aren’t useful tools; they are—the error is in mistaking the tools used for planning with the actual goals and outcomes of the execution, in emphasizing predictability over adaptability.

Organizations that adapt survive and thrive. Those that don’t disappear.

Products provide value to customers

Remember that what you’re building is a product, not a project, a business capability, not a component. Products provide tangible value to customers. A product-centered team can learn more, take a longer-term view, have increased and regular contact with the customer, and focus on measurable outcomes such as increased sales and greater customer retention—instead of plan conformance. Product-centered teams:

• Persist; they don’t dissolve and reform every few months.
• Accumulate domain knowledge.
• Build relationships with business and customers.
• Can be responsible for their own infrastructure and support.
• Can consistently participate in strategic decisions and produce business value, not just blindly churn out code according to the Plan.

Product-centered thinking, planning, and teams lead inexorably toward lean enterprise, value streams, agile teams, and hypothesis-driven development.

Throw it all away?

Should you go all #noestimates and #noprojects? Not necessarily, but the fact that such radical approaches can work in some situations is proof that traditional project management activities and mindsets are not intrinsic to software development. This evidence, and the cognitive dissonance and dissociation that project-centered thinking causes, are ample reasons to try something different.

Culture eats strategy

It is difficult to change a company’s culture to focus on products instead of projects. Project-based thinking is comfortable, provides ample space to cover your ass or hide the truth, and lets responsibility be passed along and dissipated into an unaccountable fog.

When the team is operating in project mode, it is responsible for nothing; the Plan is responsible for everything. When the team is operating in product mode, it is responsible for everything—but it can also positively influence or even control the process and thus is able to affect the outcome.

Product-centered teams have autonomy, mastery, and purpose; project-centered teams have none of these. Product-centered teams can create business value rapidly and quickly stop and pivot when pursuing a dead end. Project-centered teams must continue to the bitter end of the Plan, regardless of how hopeless they know the outcome will be.

Changing from a command-and-control, traditional, project-centered corporate culture to a lean, agile, product-focused culture is difficult, but the rewards are tremendous. One need look no further than Amazon to see how much more effective product-centered thinking can be.

Yes, your company is not Amazon, and that’s fine. But there’s nothing magical about Amazon that endows it with mystical product-centric superpowers. It consciously chose that path, and you can, too.

The price of progress reports

Yes, upper management wants status reports. It wants to know that the project is on time and under budget, that sufficient progress is being made, that everything is going according to plan, and, if it's not, that the plan is properly adjusted. This is no surprise: Upper management is often populated by Taylorists, especially accountants—in other words, by well-meaning people who just don't know any better.

It’s not about productivity

What about tracking productivity? Ah, that poor word is greatly abused, especially with regard to software development. Productivity is the wrong concept. Software is not produced by the pound; it's not dug out of the ground and refined using processes perfected over hundreds of years. It's not constructed according to physical laws and engineering principles that have been known and stable for centuries. It is created from the imaginations of multiple people, including the developers.

The very foundation of software constantly changes, and the pace of change is accelerating. The “rules” and “best practices” from even 10 to 15 years ago are now responsible for the horrific ball-of-mud monoliths that so many enterprises are struggling to escape.

Let me say that again: The best practices of the last decade—plus project-centered thinking—produced the legacy mess that developers rail against. Today's seemingly elegant architectures and practices will probably be regarded as poorly in a decade or two. That’s what progress is all about: finding better ways.

Product versus project: Business outcomes matter

If focusing on projects leads to undesirable results, try focusing on products instead. Or, if you prefer to eventually regress to mediocrity, continue to focus on project—but please stop calling yourself “agile,” “lean,” or “customer-focused.” A stack of projects is not a product any more than a pile of logs is a tree.

In fact, that’s an excellent analogy: If you want to kill a tree, saw it into a pile of logs. If you want to kill a product, saw it into a pile of projects, focus myopically on each project, change teams periodically, employ multiple hand-offs, judge the team by plan conformance, ignore the coherency of the whole, and never, ever, measure business outcomes. Soon your performance estimates will be uncannily accurate, while the best team members leave and customers begin to look for alternatives.

(With apologies to Edsger Dijkstra.)