You are here

You are here

Gated check-in system tightens DevOps feedback loops

Eitan Schichmanter Senior DevOps Architect and Leader, eBay

In DevOps, the development lifecycle usually involves developers writing their code, testing it locally on their machines, and then committing the code to a source control management system. This causes an automatic build to start, and the resulting code is evaluated to guarantee its stability, integrity, and quality.

As soon as a developer commits his or her code to the release branch, it becomes part of the release code. If the ensuing build fails due to that commit, the release branch's integrity is compromised. I crunched the numbers on the products I work with, and I found that an alarming 15 to 30 percent of builds on the release branch fail for various reasons (code, test, and environment issues), all contributing to the poor stability of the release branch.

[Also see: Feedback loops: The confluence of DevOps and big data]

The direct consequence is a severe decline in the developers' velocity, which diminishes developer trust in the build system—catastrophic for DevOps. Whenever we had a code issue, a storm of "Who's on it?" emails ensued as soon as the code was committed, as the break spread virally throughout all the development teams.

My team wasn't prepared to work like this any longer. We needed a solution.

A new way: The gated check-in system

Our demands were quite simple: Stop the code from being committed to the release branch, verify it offline on a copy of the most recent release code, and pass it back only if the verification is successful. This type of solution is called gated check-in, and it comes in several different flavors and implementations.

You can find existing gated check-in solutions out there: Gerrit, Microsoft's Team Foundation Server (TFS); Zuul; and Electric Cloud's Preflight solution are all examples, and each has its pros and cons. For example, Gerrit is widely used in the open source community, as it integrates into Git and Jenkins, and Microsoft's TFS is built in with the entire development environment.

The required characteristics and features of a gated check-in system depend on the current development flow, including but not limited to the following:

  • Non-intrusive (it doesn't change the developer's workflow) and able to run on the build server, not on the developers' machines
  • Easy for developers and DevOps to use and maintain
  • Stand-alone and development-stack agnostic
  • Lightweight and simple to enhance
  • Efficient parallel commits algorithm (addressing issues such as how to tackle concurrent commits while verification is already running)

My team and I developed a solution that we call Verigreen (verify your release code—keep it green) to address all the previous requirements. We've released it as an open source project on GitHub.

Keep learning