Persisting Data Between Steps In BitBucket Pipeline Builds

• September 03, 2020
• Technical
• Oliver

When using a build pipeline it is common to create some kind of value, like a path, container ID, auth token etc in one step which is then needed in a subsequent step but the whole point of steps is to have a temporary environment that is thrown away at the end. One option is to combine all steps into one mammoth step but there is a better way!

Artifacts To The Rescue

The solution to this is to write any values you want to keep to a file and then 'load' them in each subsequent step, but this is just moving the problem I hear you saying. Now we need to persist this file between steps, the answer? Artifacts!

We use BitBucket Pipelines for our builds and they provide a file cache that allows you to temporarily store some output from a step and then access it in another. With this and a little BASH magic we can easily keep some state between steps

This is an example pipeline build that requests an auth token in one step and then is able to access it later in other steps:

pipelines:
  branches:
    master:
    - step:
      name: Setup
      script:
      - touch environment.sh
      - echo 'ENVIRONMENT="alpha"' >> environment.sh
      artifacts:
      - environment.sh
    - step:
      name: Authenticate
      script:
      - set -o allexport; . ./environment.sh; set +o allexport
      - echo "AUTH_TOKEN=\"$(curl https://auth-server.example.org?env=${ENVIRONMENT})\"" >> environment.sh
      artifacts:
      - environment.sh
    - step:
      name: Build
      script:
      - set -o allexport; . ./environment.sh; set +o allexport
      - echo "${AUTH_TOKEN}"
      artifacts:
      - environment.sh

Here's what's happening:

1. Each step defines that it needs an artifact named environment.sh, this allows each step to access the file from previous step. It also means any changes made to the file are then available to the next step.
2. The first line of each step simply executes the file. We also set the allexport flag which means that any variables defined during execution also get defined in the context of our step.
3. Run our build commands and if we want to store some data it is appended to the artifact like so: echo 'ENVIRONMENT="alpha"' >> environment.sh

What Can You Put In The Artifact?

As we just execute the file you can put any valid BASH in there, eg:

ENVIRONMENT="production"
SITE_NAME="${REPO_NAME//./}"
IMAGE_NAME="${SITE_NAME}/${BITBUCKET_BRANCH}:${BITBUCKET_COMMIT}-${BITBUCKET_BUILD_NUMBER}"

As shown in the example above adding to the artifact can be as simple as appending a static value or it can get more involved and be a long expression though things can get interesting with quoting and escaping.