Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Optimize CI actions to use fewer minutes (/w less parallelization or more caching) #90

Closed
NevilleS opened this issue Nov 19, 2021 · 1 comment · Fixed by #92
Closed
Assignees
Labels
dev experience Enhancements to the overall DX maintenance Refactoring or ongoing maintenance work

Comments

@NevilleS
Copy link
Contributor

Right now our CI has been very helpful, but as we continue to add more tests some of the inherent inefficiencies are really adding up!

After #86 lands our checks will look something like this:
image
image

If I'm following correctly, I think that's 8 jobs, totaling 33 minutes of CI actions for one PR 😅

Of course our actual actions (tests, linters, etc.) are quite fast - but the slow part is that each of these runners needs to docker pull all the dependencies and build the images every time. There's no caching between actions either so we pay that tax on every job.

Options here are:

  1. Combine multiple jobs into a single job, so that we can reuse the Docker images across them
  2. Setup Docker caching in our CI (I think this looks right: https://evilmartians.com/chronicles/build-images-on-github-actions-with-docker-layer-caching)
@NevilleS NevilleS added dev experience Enhancements to the overall DX maintenance Refactoring or ongoing maintenance work labels Nov 19, 2021
@NevilleS NevilleS changed the title Optimize CI actions to use fewer minutes Optimize CI actions to use fewer minutes (/w less parallelization or more caching) Nov 19, 2021
@NevilleS
Copy link
Contributor Author

After some googling and reading of issue comment threads, I came across this example for enabling caching: docker/build-push-action#406 (comment)

However, we can't use that directly as our docker build commands are in the makefile and have other customizations, etc. etc. Effectively we'd need to recreate some of that cache layer in how we invoke the docker commands and it doesn't seem like that'd be particularly easy to do in a transparent way...

@NevilleS NevilleS added this to To do in Fidesops 1.1.0 via automation Nov 19, 2021
@NevilleS NevilleS self-assigned this Nov 19, 2021
@NevilleS NevilleS moved this from To do to Ready for review in Fidesops 1.1.0 Nov 19, 2021
Fidesops 1.1.0 automation moved this from Ready for review to Done Nov 19, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
dev experience Enhancements to the overall DX maintenance Refactoring or ongoing maintenance work
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

1 participant