How Accenture scaled its DevOps team

Leading a team is hard. It takes time to understand how leading differs from managing. And sometimes what seems like a good idea initially can end up being a bad idea down the road.

For instance, about four years ago—when there were about 25 people on my team—we started what we called the "DevOps Lunch," which we still have regularly. The idea is that anyone interested in attending brings his lunch to a specific area where we have a TV set up, and we run videos from conferences.

It was fun in the beginning; between seven and 10 people attended, depending on the topic.

Then I handed over the responsibility for the DevOps Lunch to someone else, who is still running them. Our team now consists of more than 120 people, and we still have the lunches, but on average only four people come. 

Sometimes, though, a person is sitting alone in front of the TV, eating his lunch while everyone else is working five meters away. He keeps doing it, hoping that one day it will magically turn around. He has ideas about how to improve DevOps Lunch. I think the answer is very simple: free food. I really believe that free food will fix the world.

There have been more bumps along the DevOps path, of course. More on that later; in the meantime, here's how DevOps leadership started for me, and lessons learned along the way.

How to Build a DevOps Toolchain That Scales

Waiting for a miracle

I joined Accenture in 2012 as a configuration management lead. However, it wasn't clear during my interview what the company expected of me. So I started practically with a blank slate and discovered that everyone was waiting for some kind of miracle.

Several months later, my team consisted of six people, and two years later it had grown to 25. At that time, DevOps was becoming very popular in the world at large, inside Accenture itself, and especially in our large enterprise clients.

But we couldn't find people with the needed skills. Accenture in the UK couldn't find people to hire, either, and its leaders came to us because they had heard we had automation skills. 

So we renamed our configuration management department the DevOps department. I realized we were not experts in DevOps, so I started reading a lot about it. I knew that we needed to stop talking about DevOps as a buzzword or an automation approach, and instead talk about doing IT more efficiently and faster.

Then we got a request from the UK to build a new Accenture DevOps platform. In addition, more clients were asking for DevOps skills specifically. The team continued to grow.

How good ideas turn to not-so-good, part 2

Among the first things we did, around six years ago, were daily standups. Initially used for one project, and later for more, they were a good practice for agile—but they were a complete failure in the long run.

The daily standups were productive for one project. But the standups didn't work for multiple projects, because people were reporting on their own projects and they didn't really care about other people's projects. Like our DevOps Lunch, it was a good idea that turned into a not-so-good idea.

So we decided each team would conduct its own daily standup.

More growing pains

In 2015, when my team consisted of about 30 people, I started to realize that the quality of our work was going down. I decided I needed some help, so I created the subject-matter expert (SME) role. These were senior people who would interview their colleagues and then talk about the different DevOps projects.

With this, we began creating a continuous improvement–oriented community using a bottom-up approach. We ran monthly sessions with the SMEs, who would assess the DevOps maturity in our projects, among other tasks.

But about six months ago, we began talking about the issues we were having with this approach. In one hour, the SMEs came up with 20 or 25 issues they would like to see improved.

That goes back to the examples of those standups and the DevOps Lunches: Any process that worked well two years ago is probably not going to fit today's requirements.

Another issue was my shifting time sinks. When there were 25 people on the team, everyone knew they could come to me with questions. But when there were 50, I was basically sitting on Skype and email all day answering questions from team members.

When all else fails, head to Thailand

So I broke the team into seven subgroups, or guilds, and created the evangelist role. Each evangelist led a guild, and the team members could talk to their evangelists before they came to me with questions. The evangelists met for an hour once a week to talk about internal processes and any problems people were having.

It worked very well, and everyone was learning from one another.

However, the number of team members increased from 50 to 100 in 18 months, and that meant a lot of learning for me. The idea of evangelists was great. Still, I was getting more and more clients.

One day last September, I was on holiday, standing in a very long line at an entertainment park in London. I was talking to myself, checking my work inbox, and I realized I was going back to work the next day but nothing had changed much. I was still spending too much time on operations.

So without confirming with anyone, I booked a six-week vacation in Thailand, beginning in January of this year.

I had three months until my vacation to come up with a new idea. And I did. I created a new role I called a "godfather"—a technical person from my team who could deal with staffing questions and help the other team members with project-related questions.

This was an unofficial role, since there was still an official project manager. Every Friday we had one-hour sessions to discuss the role and share experiences from the previous week. This worked well, and people showed great interest in being on the other side of the fence. I was able to leave for vacation in peace.

During my vacation in Thailand, I realized that most of my time had been taken up talking to those evangelists and godfathers and DevOps devotees—other people involved in non-project, non-business roles. They were all still very important, but they were taking a lot of my time asking questions, about vision and strategy, for example.

Delegate, delegate, delegate

One day during my vacation I came up with a simple answer—I needed other people to deal with this. So I sent a note to my best people about what we would do when I got back. I decided to create an evangelist lead, a godfather lead, and a devotee lead.

And that’s what I did in February when I returned. I assigned leads for those roles so I could finally focus more on strategy and a few other new responsibilities I was given in the company. The three leads were leading around 100 people in the seven guilds.

When I first invented the seven guilds, there were 50 team members, so the biggest guild had 12 people, and the smallest one had five; they were created based on people's interests.

Now that we were over 120 people, there were almost 30 people in some of the guilds. However, 12 is really the maximum number to ensure that people feel as if they belong to one team.

Yet more changes

I felt something was wrong, but I didn't have time to figure out a solution. I didn't want to create more guilds and take team members away from their evangelists, because the evangelists had invested so much energy in their people.

At one point, though, I understood the central issue: People's interests change. Two years ago, for instance, a person may have joined a guild around containers and today is still in the same guild. Even though he’s not asking to change guilds, his interests in technology may have changed and he no longer enjoys being part of that container guild.

So I came up with a new concept: chapters, which are similar to Spotify chapters but implemented a bit differently. Simply put, we are taking skills out of the guild equations and allowing people to choose which chapter they want to belong to. They can even join more than one chapter, and choose whether to lead it.

I think we have a good idea. And that’s what I will be talking about at this year's All Day DevOps conference.

Lessons learned

It's been quite a ride! Throughout this process I’ve discovered that:

  • If something's worked for two years, most probably it won't work for much longer, mainly because of growth and because the world changes. If we have a process in place for more than two years, we should get together and discuss what's wrong with it and figure out what we should improve.
  • Things need to be documented, but that's never enough. They also need to be communicated properly, which is hard.
  • It’s fine to fail, but only if a person gets back on his feet and honestly talks about it—i.e., this is the failure, this is what I learned. There will always be people who decide they don't want to do something, and just recognizing that is fine. Leaders shouldn't push those people into roles. I've learned to accept it and understand that they will be good at other things.
  • It's super hard to take a step back and look at how things are. I do that when things are just too much, which is typically too late. But it's better than having a breakdown or leaving a company.

Learn from the journey

Leading a team of 125 is hard. It takes time to understand how leading differs from managing. My best DevOps advice is to expect bumps along the way and stay open to changing things up; what works now will likely not work two years from now.

Hopefully, after my talk, readers will get a few new ideas about what they can do next in their teams.

For more from Uldis Karlovs-Karlovskis, stream his talk, "How to Lead a DevOps Team of 125 at a Fortune 100 Company," at the All Day DevOps online conference on October 17. For the full list of sessions, see the agenda at the All Day DevOps 2018 online conference website. Registration is free. All presentations will be live-streamed, and available on YouTube following the conference.

Topics: DevOps