Use CI Caching during Interop Tests #31
Comments
|
A few notes:
|
|
Notes about current stats: |
|
Related PR: #29 |
|
Run 1: 16 minutes https://github.com/laurentsenta/test-plans/actions/runs/2965877583/attempts/1 the cache for rust is
|
|
At the moment I am investigating if images are shared between tests and if we can have one 4GB cache for every test. It requires a few changes to testground, but basically we'd have: testground eval -f ./mycomposition --output ./evald-composition.toml
# output the composition with templates & other information eval'd
# we export the `buildKey` for every group here.
# for every group in the evald-composition
# load `cache/test-hash(group.buildKey)-hash(test-files)/images`
# the list of docker images required by this build (found using labels)
# load `cache/docker-{imageId}/file.tar.gz`
# contains the `docker save` for that one image
load-cache ./evald-composition.toml
testground build -f ./evald-composition.toml --output ./built-composition.toml
# build the artifacts, & exports artifact ids too
testground run -f ./build-composition.toml
# run the test, thanks for the artifact Ids parameters we don't even hit docker cache
# for every group... do the opposite save
# do not upload docker images that already exists in the cache.
save-cache ./evald-composition.tomlNote we have programmatic access to github cache via: https://github.com/actions/toolkit/tree/main/packages/cache#usage Questions:
Rough planOnce we know if the cache is a reasonable long term solution, the plan is:
|
|
Note: at the moment go + rust and just go / just rust tests are not reusing the caches because we use a different structure (one with |
|
the cache is reused between tests if we tweak the other composition files to reuse the same parameters. we can't cache per image because there are a lot of duplicated layers in that case (gzip is probably less efficient here). I wrote a script to test that assumption and killed it after generating 53GB of images. Question:
Alternative:
=> exciting options but too demanding for now. |
Todo
|
|
Notes:
|
|
layers in a go image: layer in a rust images |
|
Large test experiment: second run: https://github.com/laurentsenta/test-plans/runs/8159145324?check_suite_focus=true |
|
the temporary branch is master...laurentsenta:test-plans:feat/docker-caching (it's a really rough draft). Draft notes, will complete on Monday:
|
Docker + Ideal Caching:
Interop Practical Caching Notes:
Follow-Ups
|
Description
Running the ping interop test takes ~1min30.
Building Testground takes ~5min.
Building the Testground instances takes between 15 and 25 min.
This means 2 things:
Path for improvements
Cache build steps
I am investigating this at the moment because I believe this is where we get the most improvement:
It's not trivial:
LABELSin our Dockerfile, I'm investigating this at the moment.Cache build artifacts
At the end of the build step, it is possible to retrieve the build artifacts (docker images) produced for each group in the composition file. We could send a "pre-filled" composition file to Testground and save the part that rebuild artifacts.
The issue here is: how do you know whether an artifact needs to be rebuilt?
Even for "old" releases (like go-libp2p v0.11), we might change a test in ways that require to rebuild the test.
Do not rebuild Testground on every test
This would save around 4 min of building go & docker images. But we wouldn't get the stability benefits of NOT downloading the go & rust images + packages.
Notes
Current times
(measured with a single run, but these Github CI timings are pretty consistent)
rust
go
rust + go
rust
go
rust + go
The text was updated successfully, but these errors were encountered: