New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ci: improve install time iteration 2 #16638
Ci: improve install time iteration 2 #16638
Conversation
In case it change between yarn version 3 -> 4.
@@ -1,11 +1,17 @@ | |||
nodeLinker: node-modules | |||
|
|||
enableGlobalCache: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some might wonder, why it's suddently needed. Cause afterall it's overridden in the install action.
Context:
- By removing compression... local archives won't be zipped. That's what we want on CI... on local it's okay (more disk space, let's say it's fine). But if enableGlobalCache is true, downloaded archives won't be pruned... and over time the local installs will gros fast. See my answer in Performance opportunity for yarn 3 cache actions/setup-node#325 (comment).
PS: enableGlobalCache might be default in upcoming yarn version, so we align for the future as well)
plugins: | ||
- path: .yarn/plugins/@yarnpkg/plugin-interactive-tools.cjs | ||
spec: '@yarnpkg/plugin-interactive-tools' | ||
|
||
defaultSemverRangePrefix: '' | ||
|
||
compressionLevel: 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the zip isn't compressed. More work for action/cache restore / network / decompress (zstd detects when it can compress or not so... now the *.zip files will be compressable by action/cache -> slower but more compressed too)
For the curious:
My previous benchmarks +/- based on https://github.com/belgattitude/nextjs-monorepo-example
Speed
CI Scenario | Install | CI fetch cache | CI persist cache | Setup |
---|---|---|---|---|
yarn4 mixed-compression | ±42s | ±3s | (±9s) | 0s |
yarn4 no compression | ±34s | ±4s | (±6s) | 0s |
pnpm8 | ±19s | ±8s | (±29s) | 2s |
Cache budget
CI Scenario | Fetch cache | Persist cache | Size |
---|---|---|---|
yarn4 mixed-compression | ±5s | ±6s | 220MB |
yarn4 no compression | ±8s | ±9s | 180MB |
pnpm7 | ±14s | ±16s | 273MB |
More metrics in https://github.com/belgattitude/compare-package-managers
@@ -1,11 +1,17 @@ | |||
nodeLinker: node-modules | |||
|
|||
enableGlobalCache: false | |||
|
|||
nmMode: hardlinks-local |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one is not totally required. We'll see the ultimate choice made by upcoming yarn 4. But I'd say... keep it there explicitly (in yarn 3 it's the default).
I'm open to remove this too (anyway the install action explcitly overrides it to 'hardlinks-local')
@belgattitude I've merged your other PR - could you update this one with |
# Conflicts: # yarn.lock
@gu-stav done, but I'll have to double check how I've impacted the lock during the merge. So consider this more a wip till I'm sure nothing have been silently updated. BTW github action seems down... so let's move slowly |
# Conflicts: # yarn.lock
@gu-stav I keep this note as a reminder (about something unrelated to this P/R) one thing that I realized also is that the deps vscode/sqlite3 is consuming time and does not seem to complie in CI anyway. I'll do a cost/benefit P/R later on. Not sure of the value in the getting-started example
See the full run https://github.com/strapi/strapi/actions/runs/4946607251/jobs/8845313813 Impact is important and does not work very well in ci's (the build will fail) Myself I would remove this dep in the example. Value is limited if you're installing better-sqlite3 in the example https://github.com/strapi/strapi/blob/main/packages/core/database/lib/connection.js#L38 // NOTE: this tries to find the best sqlite module possible to use
// while keeping retro compatibility
return (
trySqlitePackage('better-sqlite3') ||
trySqlitePackage('@vscode/sqlite3') ||
trySqlitePackage('sqlite3')
); Or keep the examples outide of the main workspace (even better). Many options possibles. The suggested P/R is there #16693 |
Would be awesome to have something labelling the status to run the full test suite ❤️ |
Next iterations will be But I'm confident that we can reach a full test on < 45 minutes (vs +/-1h30 before iteration 1) soon. And more ideas to come. |
# Conflicts: # yarn.lock
…n-2' into ci-improve-install-time-iteration-2
Got the measurements now. setting the compressionLevel to 0 only saves 5-20s per install. With CE there's around 10 steps. (the saving is 1min - 3minutes only per push/run) So I'm not sure. To keep it short, Trade-offs:
So it all depends... Wdyt ? For me this PR can be merged safely... and in my experience compressionLevel:0 is a good thing, cause more deps you add, more gain you get (till this lands nodejs/node#45651 but it's a long way). PS: next iterations: #16694 and/or #16721 are part of the serie too. |
@belgattitude it seems possible to use env variables in the yarnrc file. Could we set the compression level only in a CI environment? |
Not really... as the lock contains checksums. Whenever you change compression they are recalculated. It's possible to hack of course... That's actually how I speed up docker images build:
but I don't recommend cause right now it changes the yarn.lock as well. so you'll need to rollback the change after install. for docker it's fine (divide install by 2) But that's something I'll propose to change or clarify in upstream yarn. Will PR next week and see how it's received (cause I'm not sure) |
@belgattitude thanks. I'll do some local testing tomorrow, but I think the size difference wouldn't be a huge problem. @joshuaellis @Convly @innerdvations what do you think? |
Compression level I'm not so sure about. It seems like a fairly minimal improvement compared to the tradeoffs. Are the next iterations dependent on it or can it be left alone for now? |
@innerdvations seems okay to me. I'll move it to the later iteration and when I can measure the impact based on final round about installs. So I propose this order
Then I'll measure the new baseline and adapt That's the easiest way for me Thanks |
What does it do?
Improve ci time by tackling install, part of
See #16581 and the vision: #16581 (comment)
Context
In the first iteration:
This second iteration will be more open to discussion and re-assessing the previous choices based on this new situation
Why ?
Cause in the current state it brings modification to the yarn.lock and have an impact on local disk usage.
But there's two ways I can do it
I'll try to document in the files for you to get a clear idea.
But I'm waiting first for iteration 1 to be merged (will be clearer to follow).
And as in the previous iteration, I'll need
Next steps
There's even more to do... but after those I'll have to request/coordinate with the upstream yarn codebase and if it works the cache of install-state/node_modules cache will make sense to be enabled. But that's a longer road ;) I like 🌳
Why is it needed?
🌳