Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Used cached Rust registry for conda builds #1167

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

charlesbluca
Copy link
Collaborator

Hopefully this should speed up conda builds on PRs and commits

@codecov-commenter
Copy link

codecov-commenter commented May 31, 2023

Codecov Report

Merging #1167 (b2c17d5) into main (d2e43ef) will increase coverage by 0.11%.
The diff coverage is n/a.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main    #1167      +/-   ##
==========================================
+ Coverage   81.46%   81.58%   +0.11%     
==========================================
  Files          78       78              
  Lines        4403     4403              
  Branches      798      798              
==========================================
+ Hits         3587     3592       +5     
+ Misses        640      631       -9     
- Partials      176      180       +4     

see 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Comment on lines 6 to 11
if [[ -v GITHUB_WORKSPACE ]]; then
if [[ ! -d "$GITHUB_WORKSPACE/dask_planner/target" ]]; then
mkdir "$GITHUB_WORKSPACE/dask_planner/target"
fi
ln -s $GITHUB_WORKSPACE/dask_planner/target $CONDA_PREFIX/../work/dask_planner/target
fi
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this to the script makes it so we should now be sharing work between builds on the same job (i.e. if we're building all packages in one conda build command intermediate compilation only needs to be done once); however, since each conda build command generates its own unique directory (e.g. /usr/share/miniconda3/envs/test/conda-bld/dask-sql_1685500788622), this work can't be shared between jobs because the source path is used as part of detecting a cache hit.

Going to explore if we can either:

  • Configure conda build to use a reproducible work directory on every run
  • Configure the Rust caching to cache on relative path versus absolute
    cc @ayushdg in case you've played around with the Rust caching at all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants