Businesses that successfully used agile practices to innovate in software development are applying those same principles to business operations, shedding layers of middle management in the process. The result? More innovation.

How agile is killing management, and boosting productivity

In a speech given at MIT entitled Leading by Omission, Semco Partners' CEO, Ricardo Semler, described a textile mill with a red brick building that had a canteen, looms, an operator at every station, a supervisor for every few operators, and a hierarchical management structure that extended all the way up to the owner. That business, which existed in 1633, was anything but agile.

Get 40-Page ReportNOW AVAILABLE: State of Performance Engineering 2015-16

Four centuries later, much of the world has gone from semi-literate monarchies without indoor plumbing to flourishing democracies, instant worldwide communication, and ambitious plans to create a colony on Mars. But the basic corporate structure remains the same.

It doesn't have to be that way. On his first day running Semco, Semler fired two thirds of the managers, eliminated all secretarial positions, and experienced no negative financial impact to the company. Now, businesses that have successfully used agile practices to innovate in software development are beginning to apply those same principles to business operations, shedding massive layers of middle management in the process.

The feudal corporate hierarchy

To put today's problems in context, it's useful to understand the roots of the problem. In the English feudal era you had rulers who owned land, had almost absolute power, and allowed others to use the land and the lord's power in exchange for goods and services. The hierarchy looked like this:

A typical feudal command-and-control structure

Allegiances were local because you only saw those you worked with on a day-to-day basis. Society suffered from poor internal communication due to poor literacy, and there were no markets in the modern sense because you didn't sell your goods or services to your lord: he claimed them. What's more, you didn't question your place because this was all you knew.

Fast forward to today. Here's the structure of the modern corporation:

Command-and-control from CEO to serfs

Here again, allegiances tend to be local because all you see are those you work beside on a day-to-day basis. There's poor internal communication because people aren't encouraged to get to know other departments, and there are no internal markets because you don't sell your goods or services to other groups: they claim them. If you can get an XML data feed from Team A at a cheaper price than from Team B, that's just too bad if Team B has been ordered to provide it. What's more, you tend not to question this corporate structure because this is how things are done.

Many large corporations exist as feudal societies sped up by an order of magnitude. Births and deaths are replaced by hiring and firing. In the past, bad-mouthing the king got you executed; today, bad-mouthing the CEO gets you fired. In both cases you do what you're told and turn the product of your labor over to your superior.

These corporations may also be as inefficient as feudal societies, with management layers that have become unsustainably large.

The innovation dilemma

Big businesses struggle to innovate, in part because of bloated management structures. If you're a middle manager in a large corporation, you don't need to do well to keep your job; you simply need to not do poorly.

A recent paper, The Dark Side of Analyst Coverage: The Case of Innovation, zooms in on the underlying problem: The more that analysts cover a company, the less likely it is that the company will innovate, because if it does, its share price might drop. For upper management, strong share prices are the measure of success, and the key to better compensation and job security, while poor share prices lead to hostile takeovers or bankruptcy.

Propping up share prices means not scaring the shareholders, but innovation carries an inherent risk that that will happen. Innovation, a long-term strategy, is at odds with propping up share prices, a short-term strategy. On the plus side, large corporations tend to have brand recognition, mature products, and financial reserves that give them more flexibility than smaller companies. Nonetheless, they're usually not seen as innovators.

Bottom line: Managers are not producers

Large corporations have huge numbers of managers whose job is to guide producers rather than be producers. But companies that have applied the principles of agile to management favor products over process, and therefore favor software engineers over managers. Scrum, the most popular agile system today, focuses on product owners, teams, and scrum masters?not managers. With self-directed teams, many agile companies find themselves wondering why they should bother with managers at all.

In 2012, a leaked copy of gaming company Valve Corp.'s Handbook for New Employees revealed that it has no managers. Valve Corp. is a big company, with greater profits per employee than Microsoft or Apple. However, they're not alone in their "managerless" success. Github, the top choice for software developers who want to share and collaborate on their projects, has fought for years to not have managers. Basecamp, Medium, and many other successful companies are either getting rid of managers or never had them in the first place. What's going on? How can these companies function without managers? Four reasons:

  • These tend to be private companies that aren't at the mercy of quarterly share price reports.
  • They're usually agile companies with a focus on reducing the cost of changing requirements rather than slavishly following rigid, long-term plans.
  • Their employees often get to choose what they want to do instead of being told what to do. (When you know the success of the company is dependent on your choices, you're often more careful in the choices you make.)
  • They tend to be technology companies that leverage the multiplier effect of reducing the cost of information, leading to greater profits per employee.

This trend has its roots in the dot-com boom of 1997. Prior to that time, companies enjoyed a relatively slow pace of innovation because most competitors were in the same position. Suddenly, companies faced new business opportunities, previously unknown products, and a pace of innovation that people had never seen before. While some new companies were creating products faster, others were innovating the business faster, too. If you have a solid product and can reduce your overhead by eliminating management, you have a competitive advantage.

And there it is.

The middle is falling away

The move toward managerless companies is here to stay. This trend mostly affects middle management: the C-suite roles will remain, but top managers will increasingly allow the real producers?the teams?to run their own affairs.

One of my larger clients, after completing my agile training course, began eliminating management positions and found that the company was just as productive afterwards. This trend toward eliminating middle management is slowly gaining steam, and it's working, although most of the success stories have been privately held companies. It's impossible to say what will happen with publicly traded companies in the long term, but as competitors succeed in reducing costs and improving productivity, they may be pushed into it. Thanks to agile best practices, the end of the feudal corporation may finally be in sight.

Get 40-Page ReportNOW AVAILABLE: State of Performance Engineering 2015-16
Topics: AgileApp Dev