Using Dhall To Manage GitHub Actions Workflows

Alvin Tang 2024-06-04

Technology

In my previous post, I talked about how we use Github Actions to automate our workflows. As promised, I will show how we use Dhall to manage our GitHub Actions files.

Don’t Repeat Yourself

“Don’t Repeat Yourself” (DRY) is a programming principle that reduces repetition in code. As we created more actions in a growing number of repositories, we noticed that there were a lot of repeating steps, jobs, and even whole workflows. Github Actions has features to make workflows DRY such as composite actions and reusable workflows. However, these are useful only in certain situations:

Composite actions are useful for running a sequence of steps e.g. Docker setup, login, then fetch an image.
Reusable workflows are useful for running whole jobs e.g. building and testing a Ruby on Rails repository

Github Actions does not have an easy way to reuse steps or small groups of steps. Dhall gave us that flexibility.

Life before Dhall

Every time we create a new repository, workflow files are also added. The easiest thing to do is copy a workflow file from another repository and edit it. Since the workflow has to be specific to the repository, some steps have to be modified. Eventually, this leads to repetitiveness and consistency problems. The workflows look similar but not the same. We also encountered template format errors such as misaligned tabs and missing required keys.

As the number of repositories grows, so do the workflows. We needed a more reliable way of creating new workflows and maintaining old ones.

Enter the Dhall configuration language

Dhall is a programmable configuration language. It creates JSON and YAML files with the benefits of programming principles such as types, let expressions, imports, and functions. I won’t go into detail about all features of Dhall. The documentation provides an overview and some tutorials. In this post, I will just look at some of its features that help us manage our YAML files.

`github-actions-dhall`

We did not re-invent the wheel with Dhall and GitHub Actions. There’s already an open-source project for it in GitHub. We started with that and contributed along the way whenever we thought it was useful.

Let’s start with a simple Dhall file to write a workflow that builds Node.js. We want to produce the GitHub Actions workflow below:

name: Node.js CI
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '21'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build

The Dhall file to produce the YAML above:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let checkoutStep = GitHubActions.Step::{
    , name = Some "Checkout"
    , uses = Some "actions/checkout@v2"
    }

let buildJob = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Step,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "21"
          })
        },
      GitHubActions.Step::{
        , name = Some "Install Dependencies"
        , run = Some "npm install"
        },
      GitHubActions.Step::{
        , name = Some "Run Build"
        , run = Some "npm run build"
        }
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildJob }
    }

in workflow

In this file, we define several blocks using the let expression. First, we import github-actions-dhall. github-actions-dhall has defined types for a step, job and workflow. We use Dhall’s record completion feature so we only fill in values that are not default. You can check the repository for the default values.

To produce a YAML file from Dhall, run the dhall-to-yaml executable:

dhall-to-yaml --file workflow1.dhall > workflow1.yaml

You’ll notice that Dhall produces a slightly different YAML file:

jobs:
  buildJob:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '21'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build
name: Node.js CI
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

Dhall orders the keys alphabetically. This takes some getting used to but this will work just the same. YAML doesn’t care about the order of keys.

The Dhall file is still longer than the YAML file, right? Let’s make the workflow a bit more complex by creating two jobs: one that builds node 21, and one that builds node 20. From the workflow file, we notice that only one step needs to change, actions/setup-node@v2. We need to change this to use node-version 20. In a YAML file, this means we copy the whole job then replace the value for that step only. In Dhall, I can refactor the common steps and reference the corresponding let expressions in the jobs that need it:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let checkoutStep = GitHubActions.Step::{
    , name = Some "Checkout"
    , uses = Some "actions/checkout@v2"
    }

let npmInstallStep =
      GitHubActions.Step::{
        , name = Some "Install Dependencies"
        , run = Some "npm install"
        }

let npmRunBuildStep =
      GitHubActions.Step::{
        , name = Some "Run Build"
        , run = Some "npm run build"
        }

let buildNode21Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "21"
          })
        },
      npmInstallStep,
      npmRunBuildStep
      ]
    }

let buildNode20Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "20"
          })
        },
      npmInstallStep,
      npmRunBuildStep
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildNode21Job, buildNode20Job }
    }

in  workflow

The Dhall file is getting bigger. We can utilise the import feature to break this file into multiple smaller files. Create a steps.dhall file and let’s put all our step definitions there:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let checkoutStep = GitHubActions.Step::{
    , name = Some "Checkout"
    , uses = Some "actions/checkout@v2"
    }

let npmInstallStep =
      GitHubActions.Step::{
        , name = Some "Install Dependencies"
        , run = Some "npm install"
        }

let npmRunBuildStep =
      GitHubActions.Step::{
        , name = Some "Run Build"
        , run = Some "npm run build"
        }

in { checkoutStep, npmInstallStep, npmRunBuildStep }

The variables inside in { ... } are exported. We will then import the steps.dhall file in our original Dhall file:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let Steps = ./steps.dhall

let buildNode21Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "21"
          })
        },
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let buildNode20Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = "20"
          })
        },
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildNode21Job, buildNode20Job }
    }

in  workflow

We can do the same exercise for the job expressions to make it even shorter. For now, let’s keep it this way.

Now how about the Use node.js step? They look similar and only differ in the node version. In Dhall, we can write functions to handle this. We create a function useNodeVersionStep and pass a nodeVersion argument of type Text and it outputs a GitHubActions.Step.Type. Then we call this function where we want to add a step, then pass the node version that we’d like to use:

let GitHubActions = https://raw.githubusercontent.com/regadas/github-actions-dhall/HEAD/package.dhall

let Steps = ./steps.dhall

let useNodeVersionStep
    : Text → GitHubActions.Step.Type
    = λ(nodeVersion : Text) →
        GitHubActions.Step::{
        , name = Some "Use Node.js"
        , uses = Some "actions/setup-node@v2"
        , `with` = Some (toMap {
          , node-version = nodeVersion
          })
        }

let buildNode21Job =
    GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      (useNodeVersionStep "21"),
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let buildNode20Job = GitHubActions.Job::{
    , runs-on = GitHubActions.RunsOn.Type.`ubuntu-latest`
    , steps = [
      Steps.checkoutStep,
      (useNodeVersionStep "20"),
      Steps.npmInstallStep,
      Steps.npmRunBuildStep
      ]
    }

let workflow = GitHubActions.Workflow::{
    , name = "Node.js CI"
    , on = GitHubActions.On::{
      , push = Some GitHubActions.Push::{
        , branches = Some ["main"]
        }
      , pull_request = Some GitHubActions.PullRequest::{
        , branches = Some ["main"]
        }
      }
    , jobs = toMap { buildNode21Job, buildNode20Job }
    }

in  workflow

The last three Dhall files that we made produce the same workflow YAML files:

jobs:
  buildNode20Job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '20'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build
  buildNode21Job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: "actions/checkout@v2"
      - name: Use Node.js
        uses: "actions/setup-node@v2"
        with:
          node-version: '21'
      - name: Install Dependencies
        run: npm install
      - name: Run Build
        run: npm run build
name: Node.js CI
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

How we use Dhall

Now we have a way to refactor common steps, jobs, and also create functions. We decided to put all our workflows in one repository and refactored them to make it more reusable. We have a folder structure like below:

dhall - contains all Dhall files
- deps - contains dhall dependencies such as github-actions-dhall
- libs - contains common Dhall definitions like functions and types
- steps - contains reusable steps or groups of steps
- jobs - contains reusable jobs
- composite - composite actions Dhall files with one directory per composite action
- workflows - workflow Dhall files with one directory for each repository managed
composite - the composite action YAML files with one directory for each composite action
workflows - the workflow YAML files with one directory for each repository managed

Generating the YAML files from the Dhall files is straightforward. We opted to use a simple Bash script instead of sophisticated build tools such as make since we don’t need to track dependencies.

#!/usr/bin/env bash
set -eou pipefail

echo -n "Cleanup files before generating..."
rm -rf composite/
rm -rf workflows/
echo "Done!"

echo -n "Generating composite actions..."
for COMPOSITE in $(ls dhall/composite)
do
  COMPOSITE_NAME=${COMPOSITE%.dhall}
  mkdir -p composite/$COMPOSITE_NAME
  dhall-to-yaml --file dhall/composite/$COMPOSITE >> composite/$COMPOSITE_NAME/action.yml
done
echo "Done!"

for REPOSITORY in $(ls dhall/workflows)
do
  REPOSITORY_NAME=${REPOSITORY%.dhall}
  echo -n "Generating workflows for $REPOSITORY_NAME..."
  mkdir -p workflows/$REPOSITORY_NAME
  for WORKFLOW in $(ls -p dhall/workflows/$REPOSITORY/ | grep -v /$)
  do
    dhall-to-yaml --file dhall/workflows/$REPOSITORY_NAME/$WORKFLOW >> workflows/$REPOSITORY_NAME/$WORKFLOW_NAME.yml
  done
  echo "Done!"
done

This will generate all our workflow files from the Dhall files. We then propagate these workflow files to their respective repositories using what else? GitHub Actions of course! That merits another post in the future.

How Dhall helped us

Dhall has provided us with flexibility and consistency in managing our GitHub Actions workflows.

Creating and updating workflows became less trivial because of importing and functions. Reusability of code means we are maintaining fewer lines of code.
It has helped us make fewer mistakes because of its type-checking. Something we’ve done by accident before in YAML is to delete a line and end up with completely broken YAML. In the example below, if you only delete the line with c: you accidentally put d: into a:. This is an easy mistake to make when a: and c: are both multiple pages long.

a:
  b: "b"
c:
  d: "d"

No more YAML formatting errors since the files are autogenerated. We commit YAML files alongside Dhall files, and mark them as generated with .gitattributes. This lets us fearlessly refactor because it will show no diffs in the YAML files.

If your GitHub Actions files are getting unwieldy, I suggest you start looking into Dhall to make your life easier!