DRYing GitLab CI/CD Pipelines
By Dennis D. Kirkpatrick, Lead Platform Engineer
A short time ago, a tweet thread streamed through my twitter feed between @mikebroberts and @cartwrightian lamenting coding with CloudFormation yaml.
Ian’s assertion is true. We need strong languages for infrastructure as code that support things like reuse and inheritance, just as true coding languages provide those features. Otherwise, it’s all too easy to end up with a labyrinth of complex, repeated configuration.
Working most recently on GitLab CI/CD Pipelines, I was faced with a similar dilemma. If written out in a WET (Write Everything Twice) procedural manner, YAML files can get exceedingly long and tedious with repeated portions, leading to code fragility. So, how do you define readable and durable code in your CI/CD pipelines?
Getting WET
To examine how to create DRY (Don’t Repeat Yourself) GitLab CI/CD pipelines, we’ll first look at a WET example.
Let’s assume that we have a service that is comprised of three microservices: auth, ui and app. Each microservice build is a docker build and has a similar suite of tests run against the docker image with some individualized rules and conditions from the associative build job. We’ll shorten the example by deploying into a single staging environment where additional testing will be handled outside our pipeline.
Removing some multi-line logic and using natural language to simplify the illustratration, the following GitLab CI/CD example shows how the pipeline will look without reusable components.
stages
— build_svc_image
— test_svc_image
— deploy_image
build_auth_image:
stage: build_svc_image
only:
refs:
— develop_branch
changes:
— path/to/base/Dockerfile
— path/to/base/version.txt
variables:
BUILD_PARAMETER: “some value”
BUILD_PARAMETER_TOO: “another value”
BUILD_PARAMETER_AGAIN: “yet another value”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
rules:
— first condition that dynamically provides attributes to the job
— second condition that dynamically provides attributes to the job
— third condition that dynamically provides attributes to the job
script: multiline script(s) to build and push auth service Docker imagetest_auth_image:
stage: test_svc_image
variables:
TEST_PARAMETER: “some value”
TEST_PARAMETER_TOO: “another value”
TEST_PARAMETER_AGAIN: “yet another value”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
rules:
— first condition that dynamically provides attributes to the job
— second condition that dynamically provides attributes to the job
— third condition that dynamically provides attributes to the job
image:
name: “registry.example.com/my/auth_image:latest”
entrypoint: [“/bin/bash”]
script: multiline script(s) to setup and run tests on auth service Docker image
dependencies:
— build_auth_imagebuild_ui_image:
stage: build_svc_image
only:
refs:
— develop_branch
changes:
— path/to/ui/Dockerfile
— path/to/ui/version.txt
variables:
BUILD_PARAMETER: “some value”
BUILD_PARAMETER_TOO: “another value”
BUILD_PARAMETER_AGAIN: “yet another value”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
rules:
— first condition that dynamically provides attributes to the job
— second condition that dynamically provides attributes to the job
— third condition that dynamically provides attributes to the job
script: multiline script(s) to build and push ui service Docker imagetest_ui_image:
stage: test_svc_image
variables:
TEST_PARAMETER: “some value”
TEST_PARAMETER_TOO: “another value”
TEST_PARAMETER_AGAIN: “yet another value”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
rules:
— first condition that dynamically provides attributes to the job
— second condition that dynamically provides attributes to the job
— third condition that dynamically provides attributes to the job
image:
name: “registry.example.com/my/ui_image:latest”
entrypoint: [“/bin/bash”]
script: multiline script(s) to setup and run tests on ui service Docker image
dependencies:
— build_ui_imagebuild_app_image:
stage: build_svc_image
only:
refs:
— develop_branch
changes:
— path/to/app/Dockerfile
— path/to/app/version.txt
variables:
BUILD_PARAMETER: “some value”
BUILD_PARAMETER_TOO: “another value”
BUILD_PARAMETER_AGAIN: “yet another value”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
rules:
— first condition that dynamically provides attributes to the job
— second condition that dynamically provides attributes to the job
— third condition that dynamically provides attributes to the job
script: multiline script(s) to build and push app service Docker imagetest_app_image:
stage: test_svc_image
variables:
TEST_PARAMETER: “some value”
TEST_PARAMETER_TOO: “another value”
TEST_PARAMETER_AGAIN: “yet another value”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
rules:
— first condition that dynamically provides attributes to the job
— second condition that dynamically provides attributes to the job
— third condition that dynamically provides attributes to the job
image:
name: “registry.example.com/my/app_image:latest”
entrypoint: [“/bin/bash”]
script: multiline script(s) to setup and run tests on app service Docker image
dependencies:
— build_app_imagedeploy_services:
stage: deploy
script: execute deployment of services
Do you need a towel yet? You can see a lot of repeated code in this example, even when abstracted into natural language. Imagine this file exploded out into a 500–1000+ line file. It happens!
This is extremely problematic as your code and testing changes and grows, as well as any conditional rules that will inevitably evolve due to changing business requirements. We need something more agile to match the way that we need to work. This procedural tangle of repeated code could benefit with some DRY enablement.
DRYing out the pipeline
There is a set of powerful configuration features for GitLab’s CI/CD pipelines that, used correctly, can facilitate creating more declarative, durable, and parsable pipeline configuration files. These features are:
- anchors (&) allow you mark a section of config for future reference
- aliases (*) refer back to the anchored section of config
- merge keys (<<:) allow you to insert an anchored section of config
Anchors, aliases and merge keys are actually derived from the YAML specification, which lets you easily duplicate content across a YAML document without merely repeating yourself, allowing you to follow the DRY principle (Don’t Repeat Yourself). In GitLab CI/CD’s .gitlab-ci.yml file, anchors, aliases and merge keys are used to duplicate or inherit properties, simulating functions within jobs.
From a design perspective, there are two other key elements that we need to implement to make our anchor functions work.
First, GitLab CI/CD has the include command, allowing you to export your anchor-based functions to another file that you can then import into any .gitlab-ci.yml file. This creates a YAML library of commonly used functions. We’ll use the include command in our example to import our anchor functions from our local repository.
One nice feature is that the GitLab include command can also be used to include files from another private project under the same GitLab instance, or to include a file from a different location using the full HTTP/HTTPS URL that is publicly accessible through a simple GET request (authentication schemas are not supported).
Second, we will need to implement a job-specific parameter to help generalize the anchor “functions” (e.g. $APP). Now, let’s look at our DRYer coded files.
/pipeline-functions/.docker-functions.yml
.build_docker_image: &build_docker_image # Hidden key that defines an anchor named ‘build_docker_image’
variables:
BUILD_PARAMETER: “some value”
BUILD_PARAMETER_TOO: “another value”
BUILD_PARAMETER_AGAIN: “yet another value”
rules:
— first condition that dynamically provides $APP attributes to the job
— second condition that dynamically provides $APP attributes to the job
— third condition that dynamically provides $APP attributes to the job
script: multiline script(s) to build and push $APP service Docker image.test_docker_image: &test_docker_image # Hidden key that defines an anchor named ‘test_docker_image’
variables:
TEST_PARAMETER: “some value”
TEST_PARAMETER_TOO: “another value”
TEST_PARAMETER_AGAIN: “yet another value”
rules:
— first condition that dynamically provides $APP attributes to the job
— second condition that dynamically provides $APP attributes to the job
— third condition that dynamically provides $APP attributes to the job
image:
name: “registry.example.com/my/$APP:latest”
entrypoint: [“/bin/bash”]
script: multiline script(s) to setup and run tests on auth service Docker image
.gitlab-ci.yml
stages
— build_svc_image
— test_svc_image
— deploy_imageinclude:- local: ‘/pipeline-functions/.docker-functions.yml’
build_auth_image:
stage: build_svc_image
<<: *build_docker_image # Merge the contents of the ‘build_docker_image’ alias
only:
refs:
— develop_branch
changes:
— path/to/$APP/Dockerfile
— path/to/$APP/version.txt
variables:
APP: “auth”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”test_auth_image:
stage: test_svc_image
<<: *test_docker_image # Merge the contents of the ‘test_docker_image’ alias
variables:
APP: “auth”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
dependencies:
— build_auth_imagebuild_ui_image:
stage: build_svc_image
<<: *build_docker_image # Merge the contents of the ‘build_docker_image’ alias
only:
refs:
— develop_branch
changes:
— path/to/ui/Dockerfile
— path/to/ui/version.txt
variables:
APP: “ui”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”test_ui_image:
stage: test_svc_image
<<: *test_docker_image # Merge the contents of the ‘test_docker_image’ alias
variables:
APP: “ui”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
dependencies:
— build_ui_imagebuild_app_image:
stage: build_svc_image
<<: *build_docker_image # Merge the contents of the ‘build_docker_image’ alias
only:
refs:
— develop_branch
changes:
— path/to/app/Dockerfile
— path/to/app/version.txt
variables:
APP: “app”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”test_app_image:
stage: test_svc_image
<<: *test_docker_image # Merge the contents of the ‘test_docker_image’ alias
variables:
APP: “app”
JOB_PARAMETER: “some other value”
JOB_PARAMETER_TOO: “another value too”
JOB_PARAMETER_AGAIN: “another value again”
dependencies:
— build_app_imagedeploy_services:
stage: deploy
script: execute deployment of services
Compared to the previous version, we were able to reduce the .gitlab-ci.yml lines by 30%. The design benefit is that repeated-use code is consolidated into just two reusable functions, instead of six independent procedures, a 66% reduction in functional complexity with only one added data parameter or string value. Finally, our functions are exported to an external file, making them reusable in additional pipelines as desired.
Conclusion
Anchors, aliases and merge keys are a powerful and effective combination when writing out YAML encoded pipelines in GitLab CI/CD pipeline. They facilitate the DRY principle in your YAML files, allowing you to write code once and use it many times, making readable and durable code within your CI/CD pipelines.
I recommend looking at other CI/CD tools that enable these kinds of reusable, modular YAML usage: Concourse-CI, Azure Pipelines, BitBucket Pipelines, CircleCI, and even in configuring Kubernetes objects. As with any design pattern, there are good and poor use cases. Still, as the opening tweet shared, we owe it to ourselves to continue wrestling with how to maximize the tools we use in this Everything-as-Code world.
