DevOps: Small Deployment Size and Low Work-in-Progress Drive Success

Fast and Frequent: Is There Really a Need For Speed?

Most DevOps transformations focus on speed. The primary goals are often to:

  1. Shorten the time between code being committed to code running in production(Lead Time for Changes).
  2. Deploy code to production more frequently(Deployment Frequency)

Deployment Size and WIP are What Matters

Deploying Small and Single Piece Flow

The benefits of DevOps are not derived from going fast, but from continuously pushing small amounts of code to production in a process similar to single-piece flow in manufacturing. Within limits, the smaller the amount of code pushed through a DevOps pipeline into production the better.

  • Small amounts of code are easier to develop and test than large amounts of code. Testing small amounts of code is simpler and less error prone than testing large amounts of code.
  • Small chunks of code are easier to build and deploy. The time to deploy is less because the complexity is less.
  • Monitoring and defect detection are easier. In the event a defect is released to production it is much easier to pinpoint the problem when only a small amount of code has been released. This reduces the DORA metric Mean Time To Resolution.
  • It is easier to roll back small amounts of code if there is a production defect. In practice, changes in a small deployment model are often so incremental that the customer impact of any defect is negligible. A “fix forward” approach is often used instead of a rollback.
  • It is easier and quicker to get customer and stakeholder feedback on the deployment of a small feature, and easier to implement future changes based on that feedback.
  • Constant small deliveries of value to customers increases both customer satisfaction and the developer experience of engineers.

Deploying Small is Not Enough: How High WIP Increases Decreases Team Effectiveness

The benefits of small deployments are greatly reduced if Lead Time for Changes gets so large that multiple deployments are stacked up in a DevOps pipeline. Stacked deployments means WIP increases, and when WIP increases the cognitive load on a team increases and software quality tends to suffer.

Know Where The Constraints Are

Now we understand why rapidly moving code through a pipeline (Lead Time for Changes) is so important — it reduces cognitive load by reducing WIP and also results in small deployments.

The Wrong Branching Model Can Increase Deployment Size and WIP

Gitflow vs Trunk-based Development

Typical Gitflow pattern. Image by Toptal
Typical Trunk-based workflow. Image by Vincent Driessen

The Deployment Death Spiral

Deployment Frequency is Still Very Important

To review, deployments of small amounts of code drive improvement in software quality, developer experience and customer satisfaction. A team has to deploy frequently if they want to continually deploy small amounts of code while maintaining good productivity. Frequent deployments enable small deployments which improve software quality.

Four Steps to Escape the Deployment Death Spiral

Being trapped in the Deployment Death Spiral is painful. There are no cookie-cutter solutions to this problem. Each engineering group and company are different, as are the demands placed upon them. Political considerations are different everywhere.

  • Merging new code into the DevOps pipeline just increases WIP and increases the size of the next deployment. This is exactly what a team wants to avoid. Only merge code for absolutely “must have” features or bug-fixes. This reduces WIP in the pipeline and the size of the next deployments.
  • WIP still exists in the form of features waiting to be merged into the DevOps pipeline, but for now this form of WIP is easier to manage than WIP in the pipeline itself.
  • This step reduces Deployment Size and lowers WIP.
  • This step can sometimes be done concurrently with Step 1.
  • This step can be politically challenging to implement. However, if 100% of an engineering team is always putting out fires there will never be any capacity to improve.
  • Moving engineers from writing application code to work on DevOps pipelines or automated testing reduces WIP (by not creating it in the first place) and reduces the size of the next deployment.
  • Automating the DevOps pipeline makes it run faster. This reduces WIP and reduces Deployment Size by getting code to production faster. Value-Stream Mapping can identify constraints to be removed.
  • Automating testing reduces overall WIP by getting code through the DevOps pipeline and into production faster. It can take some time to automate testing in a large application, so one good first step is to require any new feature or defect fix be tested entirely through automation. Just this one practice can make a profound difference in Deployment Frequency, Deployment Size, Change Failure Rate and WIP.
  • This step lowers WIP and reduces Deployment Size.
  • Remember that high Deployment Frequency leads to low Deployment Size. Low deployment Frequency leads to high Deployment Size.
  • Use increasing Deployment Frequency as a forcing function. If a team is deploying quarterly, challenge them to deploy monthly. If they deploy monthly, challenge them to deploy every two weeks. The team provides a list of steps to meet the Deployment Frequency goal and then implements those steps. This can be very effective at identifying waste in the entire code/build/deploy process.
  • Working on increasing Deployment Frequency also keeps engineers from adding code to the DevOps Pipeline, which reduces WIP and the size of the next deployment.
  • Keep challenging the team to reduce Deployment Size. The optimal way to do this is to deploy very small chunks of code virtually on demand. However, a team does not need to deploy on demand to see substantial improvements in application quality and overall work life.
  • Important Note: this isn’t about forcing engineers to work harder or produce more features. The amount of work produced by the team should be constant. Instead, the team is simply reducing Deployment Size by deploying the same amount of work to production more frequently.
  • This step reduces Deployment Size and lowers WIP.
  • Use cross-functional teams to design and construct a DevOps pipeline. This team’s objective is reducing WIP and reducing Deployment Size by rapidly moving code through the pipeline.
  • Fight against any manual checks or approvals. If a human can test quality, so can a machine. But a machine can do it over and again without getting bored, distracted or tired.
  • Reduce hand-offs to a bare minimum, even in an automated process. Hand-offs are weak points in any process.

A Virtuous Upward Spiral Begins

Implementing these steps can produce a virtuous upward spiral, essentially the reverse of the Deployment Death Spiral. The team’s capacity to produce quality software is increased at every step.

  • The faster a team deploys the smaller their deployments become.
  • Smaller more frequent deployments reduce WIP.
  • WIP is further reduced by automating DevOps pipelines and testing.
  • Small deployments and lower WIP lower Change Failure Rates (odds of a defect released to production). The team spends less time fixing defects after every production release and more time providing value to customers.
  • Even if a defect is released to production, it is easier to find and fix the defect because less code was released. This lowers Mean Time To Resolution.
  • An organization’s confidence in automated DevOps pipelines and automated testing increases as the quality of deployments increases. More and more testing is automated, which reduces WIP and reduces Change Failure Rate.
  • The cognitive load on the team continues to decline. They have more time and capacity to further decrease Deployment Size and reduce WIP. Software quality increases.
  • Production defects become very rare. Instead of spending engineering capacity putting out production fires, engineers are continually seeking new ways to improve their processes and delight customers.

--

--

Make Next Possible

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store