Allow to specify cache folders on Linux

[mayeut]

Many CI services if not all allow caching folders.
While this can be leveraged easily on macOS & Windows with cibuildwheel, this is not the case on Linux where builds occur in an isolated docker container.
The idea would be to add a configuration option on Linux to specify a list of folders to synchronize in the containers.

I implemented caching using an implementation detail of cibuildwheel in mayeut/cmake-python-distributions@339c4e9 which allows to reuse the cache from previous runs for the before_all step.

This might be very interesting for emulated builds where everything is slow & I could reduce CI time by 20 minutes by not rebuilding OpenSSL every time: https://github.com/mayeut/cmake-python-distributions/runs/3209785906?check_suite_focus=true

[henryiii]

Would this be best combined with feature=in-tree-build #742 ?

[mayeut]

I think the feature I'm thinking about is orthogonal although one might want to cache part of the artifacts built by a python build.

[joerick]

This sounds good to me, I think. I'm wondering how to implement/expose it. We already copy the contents of the project_dir into the container, so I'm guessing the cache you want to import is outside this dir? It could have a generalised 'mapping' option, like CIBW_CONTAINER_MAP=host_path:container_path, or it could just be a convention, like if there's anything inside project_dir/cache, it's automatically mapped into the container at /cibw/cache?

[mayeut]

so I'm guessing the cache you want to import is outside this dir?

Not necessarily, I usually keep the cache folder in the same folder as the project. The issue is thus not to get the cache in the container but rather to synchronize it back out of the container.

It could have a generalised 'mapping' option, like CIBW_CONTAINER_MAP=host_path:container_path, or it could just be a convention, like if there's anything inside project_dir/cache, it's automatically mapped into the container at /cibw/cache?

We might want to synchronize back the project_dir no matter what ? This might add some time on large projects ?
CIBW_CONTAINER_MAP seems like a sensible thing to add but it should take a list of map e.g. CIBW_CONTAINER_MAPS=host_path1:container_path1,host_path2:container_path2

[joerick]

We might want to synchronize back the project_dir no matter what ?

I'm not sure, could this be a breaking change for some people? Probably there are lots of people relying on the isolated nature of the docker build? If that's not a concern, we could consider mounting rather than copying the project dir in (1. from #466), that would provide the behaviour you need I think. We'd only have to do the 'copy back' on CircleCI.

I also wonder if in-tree-build will complicate this a bit further, where build artifacts are more likely to be shared between builds. e.g. would that be a problem if users are building some python versions on manylinux2010 and some on manylinux2014?

The issue is thus not to get the cache in the container but rather to synchronize it back out of the container.

This is interesting. I now wonder if this feature could be general enough to also cover the use case of extracting coverage reports from the container. If the above approach doesn't work, I'm wondering about something like
CIBW_CONTAINER_MAPS={project_dir}/cache,/host/absolute/path/:/container/path - where if the path is relative to the project, you don't need to specify both sides, as they are the same.

[bmerry]

I now wonder if this feature could be general enough to also cover the use case of extracting coverage reports from the container.

+1 for a feature to get build artifacts other than the wheels out of the container. I have a setup where I generate .debug files that contain debug symbols corresponding to the built shared libraries that go into the wheels. At the moment I'm abusing the fact that cibuildwheel writes the repaired wheels into /output and putting my debug symbols there as well, but I'd be happier if there was a supported way to do it.

[strayer]

I'm currently fighting with this too. I'm trying to refactor pytaglibs Github Action to use cibuildwheel to provide wheels for more platforms (especially Linux aarch64 and macOS arm64).

Currently compiling taglib in the container tages over 13 minutes on aarch64 due to qemu. If I could somehow cache that result it would speed up the wheel builds (which should run in every PR) dramatically.

The other platforms (Windows, macOS) cache a folder named taglib-install and taglib-source and work completely fine. On Linux I could imaging caching taglib-source and having subdirectories there. This would require having some kind of platform key in the container too though. cibuildwheel seems to be smart enough to only run my compilation in before_all once per manylinux/musllinux "platform", so using the platform description as a subfolder name should be enough to not influence other platforms.

Restoring the cache would work already since cibuildwheel copies the project into the container, but I'm unsure how to get the cache out… I probably could hack something using the /output folder mentioned above, but then I still need some kind of environment variable in the container to name the subfolder for the correct manylinux/musllinux variant.

Any tips would be greatly appreciated… I think this could benefit pytaglib a lot, but I can't see the maintainer accepting a PR increasing the CI build times to 30+ minutes :(

[wbarnha]

Bumping for attention as well, I need to repeatedly build the same RocksDB libraries and I'd like to be able to cache them somewhere to be used alongside cibuildwheel in order to reduce build times from 15 minutes. I think mapping select build artifacts to /cibw/cache would dramatically save me time building.

[makslevental]

@joerick are you still in support of this? If so I'd be happy to send up a PR.

[lkeegan]

This issue appears to be resolved as the root of the host file system / is now mounted as /host in the container: https://cibuildwheel.readthedocs.io/en/stable/faq/#linux-builds-in-containers