The CI Setup: GitHub Actions

Introduction

With the migration to GitHub Actions (almost) complete, this page is a quick overview of the CI setup on Kedro.

First, some important links:

• All the workflows reside in .github/workflows
• Link to GitHub Actions docs - https://docs.github.com/en/actions
• All the workflows can be seen on the "Actions" tab - https://github.com/kedro-org/kedro/actions
• Miro board with all the diagrams included on this page (Note: only accessible internally)

The Test Setup

Here's a diagram to visualise the test setup on Kedro and how all the workflows interact with each other:

Reusable Workflows

unit-test.yml

• Triggered by calls from another workflow : Called by all-checks.yml
• Inputs: os, python-version, branch
• Checks out code, installs dependencies and runs unit-tests

lint.yml

• Triggered by calls from another workflow : Called by all-checks.yml & docs-only-checks.yml
• Inputs: os, python-version, branch
• Checks out code, installs dependencies and runs linter

e2e-tests.yml

• Triggered by calls from another workflow : Called by all-checks.yml
• Inputs: os, python-version, branch
• Checks out code, installs dependencies and runs end to end tests.

pip-compile.yml

• Triggered by calls from another workflow : Called by all-checks.yml
• Inputs: os, python-version, branch
• Checks out code, installs dependencies and runs pip compile command.

all-checks.yml

• Triggered by:
  • PRs on main and develop branch
  • Push on main and develop branch
  • Called by nightly-build.yml
• Calls - unit-test.yml, lint.yml, e2e-tests.yml and pip-compile.yml as separate jobs.

Docs related workflows

docs-only-check.yml

• Triggered by:
  • PRs on main and develop branch
  • Push on main and develop branch
• Triggered when there's changes to docs, i.e. .md files or files in the docs/ folder.

docs-language-linter.yml

• Triggered by PRs with changes to docs, i.e. .md files or files in the docs/ folder.
• Runs the vale linter on the changed lines.
• Adds suggestions as GitHub PR checks comments.
• Does not block merging.
• Grammar rules used by vale are in .github/styles/Kedro

Nightly build setup

nightly-build.yml

• Runs every night at midnight UTC.
• Job kedro-main-test calls all-checks.yml to trigger all tests on the main branch.
• Job notify-main creates an issue with "main nightly build" label if there is a failure or adds a comment on the issue if it already exists.
• Job kedro-develop-test calls all-checks.yml to trigger all tests on the develop branch.
• Job notify-develop creates an issue with "develop nightly build" label if there is a failure or adds a comment on the issue if it already exists.

Note

There is no way to run a scheduled workflow on non-default branches (i.e. develop)(for now, at least!). The way this workflow runs on both branches is by passing the branch name as an input to the reusable workflows. The relevant branch is the checked out at the "Checkout code" step in the reusable workflows - unit-test/lint/e2e-tests/pip-compile.yml The nightly build workflow uses the main branch version of the workflow but the develop branch code. Some of the failures on the develop nightly build job might be because of this.

Merge Gatekeeper

merge-gatekeeper.yml

• Triggered by PRs on main and develop
• Is required to pass for the PRs to be able to be merged.
• Runs until timeout and passes if the test suite passes.

The Release Setup

Here's a diagram to visualise the release workflow of kedro and kedro-starters:

Kedro Release

check-release.yml

• Triggered by push on main
• The first job - check-version checks if the version specified on the kedro/__init__.py file is on PyPI already or not. If it isn't, that means a release should be triggered.
• The script for checking of versions is kedro/tools/github_actions/github_actions_release.py
• The second job - test-kedro runs the test suite with all-checks.yml
• The third job - build-publish builds the package, extracts release notes from RELEASE.md(using tools/github_actons/extract_release_notes.py), creates a GitHub release and a PyPI release.

release-starters.yml

• Triggered by a GitHub release publish event.
• Sends a "repository dispatch" to kedro-starters
• Triggers the release.yml workflow on kedro-starters which creates a GitHub release with the version number tag.

The Sync and Merge Setup

These workflows are required to keep the main and the develop branches in sync automatically. The diagram below illustrates the workflows and how they work together.

sync.yml

• Scheduled to run every 30 mins
• Uses the same script previously used by CircleCI (tools/github_actions/merge.sh)
• Merges the commits on main to develop if there are no conflicts
• Raises a PR if there are conflicts.
• Note: Uses GH_TAGGING_TOKEN as credentials which is a PAT that belongs to the admin.

auto-merge-prs.yml

• Scheduled to run every 60 mins
• Uses the same script previously used by CircleCI [tools/github_actions/attempt_merge_pr.sh
• Merges a PR that is open and approved from merge-main-to-develop branch to develop.
• Note: Uses GH_TAGGING_TOKEN as credentials which is a PAT that belongs to the admin.

[OUTDATED]

label-pr.yml

• Triggered by a PR opening on develop branch.
• Labels the PRs on the branch merge-main-to-develop with automerge. This is required for the auto-merge-pr.yml action to identify the sync PRs.
• NOTE: This workflow runs once the conflicts are resolved.
• The config for this action is in .github/labeler.yml

auto-merge-prs.yml

• Scheduled to run every 60 mins
• IMPORTANT: This action merges ANY PRs labelled "automerge" with merge commits enabled. Therefore, it is important to not incorrectly label PRs as "automerge"
• Will only merge PRs if the branch protection rules are satisfied.