Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create ADR for integrating cache functionality to setup-python action #247

Merged
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
93 changes: 93 additions & 0 deletions docs/adrs/0000-caching-dependencies.md
@@ -0,0 +1,93 @@
## 0. Caching dependencies

Date: 2021-10-01

Status: Proposed

## Context

`actions/setup-python` is one the most popular python's action in GitHub Actions. A lot of customers use it in conjunction with `actions/cache` to speed up dependencies installation.
See more examples on proper usage in [actions/cache documentation](https://github.com/actions/cache/blob/main/examples.md#python---pip).

## Goals & Anti-Goals

Integration of caching functionality into `actions/setup-python` action will bring the following benefits for action users:
- Decrease the entry threshold for using the cache for Python dependencies and simplify initial configuration
- Simplify YAML pipelines by reducing the need for additional steps to enable caching
- More users will use caching for Python so users will have faster builds!

We will add support for Pip and Pipenv dependency caching.

We won't pursue the goal to provide wide customization of caching in the scope of `actions/setup-python` action. The purpose of this integration is to cover ~90% of basic use-cases. If users need flexible customization, we will advise them to use `actions/cache` directly.

## Decision

- Add a `cache` input parameter to `actions/setup-python`. For now the input will accept the following values:
- pip - enable caching for pip dependencies
- pipenv - enable caching for pipenv dependencies
- '' - disable caching (default value)
- Cache feature will be disabled by default to make sure that we don't break existing customers.
- Action will try to search dependencies files (requirements.txt for pip and Pipfile.lock for pipenv) in the repository root (or relative to the repository root, if patterns are used) and throw error if no one is found.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most popular file to manage required packages for pip is requirements.txt, so we've decided to use it as default one to calculate hash for the cache key. The main problem with this file is that the requirements file format allows to specify dependency versions using logical operators (for example chardet>=3.0.4) or specify dependencies without any versions. In this case the pip install -r requirements.txt command will always try to install the latest available package version. It can lead to an increase in total build time, since restored cache will not be used if the requirements.txt file is not updated and a newer version of the dependency is available.

Unfortunately, there is no widely used lock file for the pip package manager, so we've decided to use requirements.txt file. We will add a note in the docs about possible issue and provide the cache-dependency-path input so that customer can select a different dependencies file.

Any thoughts and feedback are appreciated

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I agree with this approach 👍 I've mostly seen requirements.txt being used so I think it makes perfect sense to start off with supporting this.

- The hash of found file will be used as part of cache key (the same approach like actions/cache recommends)
- The following cache key will be used for pip: `setup-python-${{ runner.os }}-pip-${{ hashFiles('<package-file-path>') }}`
- The following cache key will be used for pipenv: `setup-python-${{ runner.os }}-python-${{ python-version }}-pipenv-${{ hashFiles('<package-file-path>') }}`. We add the python version to the cache key because the created virtualenv folder with the project name contains a copy of the python binary as a symlink to paths like `/opt/hostedtoolcache/Python/3.7.11`, so the cache can be fetched with a wrong python version. See details in the related [pull request](https://github.com/actions/cache/pull/607) in the actions/cache.
- Action will save the packages global cache:
- Pip (retrieved via pip cache dir). The command is available With pip 20.1 or later. We always update pip during installation, that is why this command should be available.
- Pipenv (default cache paths):
- ~/.local/share/virtualenvs (macOS)
- ~/.virtualenvs (Windows)
- ~/.local/share/virtualenvs (Ubuntu)
- Add a `cache-dependency-path` input parameter to `actions/setup-python`. The new input will accept an array or regex of dependency files. The field will accept a path (relative to the repository root) to dependency files. If the provided path contains wildcards, the action will search all matching files and calculate a common hash like the `${{ hashFiles('**/requirements-dev.txt') }}` YAML construction does

## Example of real use-cases

- Pip package manager

```
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
cache: pip
```

- Pipenv package manager

```
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
cache: pipenv
```
- With `cache-dependency-path`

```
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
cache: pip
cache-dependency-path: |
**/requirements-dev.txt
**/requirements-test.txt
**/requirements.txt
```


```
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.9
cache: pip
cache-dependency-path: **/requirements-dev.txt
dmitry-shibanov marked this conversation as resolved.
Show resolved Hide resolved
```

## Release process

As soon as the functionality is implemented, we will release a minor update of the action. No need to bump the major version since there are no breaking changes for existing users. After that, we will update [starter-workflows](https://github.com/actions/starter-workflows/blob/main/ci/python-app.yml) and [GitHub Action documentation](https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#caching-dependencies).