Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: improve Yarn 2 cache on CI #11781

Merged
merged 6 commits into from Aug 26, 2020
Merged

Conversation

JLHwung
Copy link
Contributor

@JLHwung JLHwung commented Jul 2, 2020

Q                       A
Fixed Issues? Add @actions/cache to our GitHub coverage workflow
License MIT

Based on example: https://github.com/actions/cache/blob/master/examples.md#node---yarn

The cache is working as expected. We can compare the Yarn fetch time between main and this PR.

main This PR
2.4m 0.91s

@babel-bot
Copy link
Collaborator

babel-bot commented Jul 2, 2020

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/27863/

@codesandbox-ci
Copy link

codesandbox-ci bot commented Jul 2, 2020

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit b990407:

Sandbox Source
babel-repl-custom-plugin Configuration

@JLHwung JLHwung marked this pull request as ready for review July 2, 2020 13:47
@JLHwung JLHwung added the PR: Internal 🏠 A type of pull request used for our changelog categories label Jul 2, 2020
@JLHwung JLHwung changed the base branch from next-8-dev to main August 15, 2020 18:58
@JLHwung JLHwung changed the title chore: add cache actions chore: improve Yarn 2 cache on CI Aug 15, 2020
@@ -3,13 +3,45 @@ name: Node CI
on: [push, pull_request]

jobs:
validate-yarn-cache:
Copy link
Contributor Author

@JLHwung JLHwung Aug 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step ensures the test jobs later will always be served with proper yarn cache.

cc @arcanis Do you think yarn 2 will ever support yarn install --fetch-only, which skips the linking stage and only fetche new packages to yarn cache folder. The idea is to run yarn install --fetch-only only if yarn.lock is changed, and the CI jobs afterward will be served with a proper yarn cache. In all it can save network requests and reduce carbon emissions.

Currently when yarn.lock is updated, the validate-yarn-cache job will introduce 20s (linking) overhead to our CI -- since we meant to update the yarn cache folder only.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it'll be in the default distribution. One problem with the v1 was that installs could be done in various slightly different ways that made the surface for unforeseen edge cases much larger. I'd like to avoid that with one standardized path as much as possible.

That being said, a plugin to do this would be trivial, so we probably can find something eventually, once we've figured out a good way to develop/host such plugins (it's already possible, but there's no comprehensive documentation).

Copy link
Contributor

@merceyz merceyz Aug 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can sort of get this behaviour if you set this specific job to install using PnP and with build scripts disabled, setting these env variable should do the trick:

YARN_NODE_LINKER=pnp
YARN_ENABLE_SCRIPTS=false

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@merceyz We opt-out of PnP because Flow does not support PnP: yarnpkg/berry#634 (comment).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, but for this specific job, validate-yarn-cache, its job is to generate a valid cache, not run Flow, so installing using PnP would do that without the cost of linking. The cache for PnP and node-modules is the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome: https://github.com/babel/babel/pull/11781/checks?check_run_id=991006280#step:6:27 The pnp linker costs only 4s compared to 20s of node-modules.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can reduce it even more if you also set YARN_ENABLE_SCRIPTS=false, you can do this since you don't need to run postinstalls to get a valid cache

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JLHwung JLHwung mentioned this pull request Aug 15, 2020
@JLHwung JLHwung force-pushed the add-github-actions-cache branch 6 times, most recently from 34bb1a4 to ee7912c Compare August 20, 2020 19:35
@JLHwung JLHwung force-pushed the add-github-actions-cache branch 3 times, most recently from e3dbdc4 to d484dbb Compare August 25, 2020 17:27
@@ -3,13 +3,46 @@ name: Node CI
on: [push, pull_request]

jobs:
validate-yarn-cache:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: maybe this should be called prepare-yarn-cache, since "validate" implies that it would fail the build if, for example, the cache is old.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JLHwung JLHwung merged commit b8f3453 into babel:main Aug 26, 2020
@JLHwung JLHwung deleted the add-github-actions-cache branch August 26, 2020 15:53
@github-actions github-actions bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Nov 26, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated A closed issue/PR that is archived due to age. Recommended to make a new issue PR: Internal 🏠 A type of pull request used for our changelog categories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants