Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add poetry caching support #281

Merged
merged 15 commits into from Mar 31, 2022
30 changes: 26 additions & 4 deletions .github/workflows/e2e-cache.yml
Expand Up @@ -21,7 +21,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.8', '3.9', 'pypy-3.7-v7.x']
python-version: ['3.9', 'pypy-3.7-v7.x']
steps:
- uses: actions/checkout@v2
- name: Setup Python
Expand All @@ -39,7 +39,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.8', '3.9', 'pypy-3.7-v7.x']
python-version: ['3.9', 'pypy-3.7-v7.x']
steps:
- uses: actions/checkout@v2
- name: Setup Python
Expand All @@ -52,14 +52,36 @@ jobs:
- name: Install dependencies
run: pipenv install numpy

python-poetry-dependencies-caching:
name: Test poetry (Python ${{ matrix.python-version}}, ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.9', 'pypy-3.7-v7.x']
steps:
- uses: actions/checkout@v2
- name: Install poetry
run: pipx install poetry
- name: Setup Python
uses: ./
with:
python-version: ${{ matrix.python-version }}
cache: 'poetry'
- name: Init pyproject.toml
run: poetry init -n
- name: Install dependencies
run: poetry add flake8

python-pip-dependencies-caching-path:
name: Test pip (Python ${{ matrix.python-version}}, ${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.8', '3.9', 'pypy-3.7-v7.x']
python-version: ['3.9', 'pypy-3.7-v7.x']
steps:
- uses: actions/checkout@v2
- name: Setup Python
Expand All @@ -78,7 +100,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.8', '3.9', 'pypy-3.7-v7.x']
python-version: ['3.9', 'pypy-3.7-v7.x']
steps:
- uses: actions/checkout@v2
- name: Setup Python
Expand Down
25 changes: 20 additions & 5 deletions README.md
Expand Up @@ -9,7 +9,7 @@ This action sets up a Python environment for use in actions by:
- optionally installing and adding to PATH a version of Python that is already installed in the tools cache.
- downloading, installing and adding to PATH an available version of Python from GitHub Releases ([actions/python-versions](https://github.com/actions/python-versions/releases)) if a specific version is not available in the tools cache.
- failing if a specific version of Python is not preinstalled or available for download.
- optionally caching dependencies for pip and pipenv.
- optionally caching dependencies for pip, pipenv and poetry.
- registering problem matchers for error output.

# What's new
Expand All @@ -19,7 +19,7 @@ This action sets up a Python environment for use in actions by:
- Automatic setup and download of Python packages if using a self-hosted runner.
- Support for pre-release versions of Python.
- Support for installing any version of PyPy on-flight
- Support for built-in caching of pip and pipenv dependencies
- Support for built-in caching of pip, pipenv and poetry dependencies

# Usage

Expand Down Expand Up @@ -209,18 +209,19 @@ pypy-3.7-nightly # Python 3.7 and nightly PyPy

# Caching packages dependencies

The action has built-in functionality for caching and restoring dependencies. It uses [actions/cache](https://github.com/actions/toolkit/tree/main/packages/cache) under the hood for caching dependencies but requires less configuration settings. Supported package managers are `pip` and `pipenv`. The `cache` input is optional, and caching is turned off by default.
The action has built-in functionality for caching and restoring dependencies. It uses [actions/cache](https://github.com/actions/toolkit/tree/main/packages/cache) under the hood for caching dependencies but requires less configuration settings. Supported package managers are `pip`, `pipenv` and `poetry`. The `cache` input is optional, and caching is turned off by default.

The action defaults to searching for a dependency file (`requirements.txt` for pip or `Pipfile.lock` for pipenv) in the repository, and uses its hash as a part of the cache key. Use `cache-dependency-path` for cases where multiple dependency files are used, they are located in different subdirectories or different files for the hash want to be used.
The action defaults to searching for a dependency file (`requirements.txt` for pip, `Pipfile.lock` for pipenv or `poetry.lock` for poetry) in the repository, and uses its hash as a part of the cache key. Use `cache-dependency-path` for cases where multiple dependency files are used, they are located in different subdirectories or different files for the hash want to be used.

- For pip, the action will cache global cache directory
- For pipenv, the action will cache virtualenv directory
- For poetry, the action will cache virtualenv directory

**Please Note:** Restored cache will not be used if the requirements.txt file is not updated for a long time and a newer version of the dependency is available that can lead to an increase in total build time.

The requirements file format allows to specify dependency versions using logical operators (for example chardet>=3.0.4) or specify dependencies without any versions. In this case the pip install -r requirements.txt command will always try to install the latest available package version. To be sure that the cache will be used, please stick to a specific dependency version and update it manually if necessary.

**Caching pip dependencies:**
**Caching pip dependencies:**

```yaml
steps:
Expand All @@ -245,6 +246,20 @@ steps:
- run: pipenv install
```

**Caching poetry dependencies:**
```yaml
steps:
- uses: actions/checkout@v2
- name: Install poetry
run: pipx install poetry
- uses: actions/setup-python@v2
with:
python-version: '3.9'
cache: 'poetry'
- run: poetry install
- run: poetry run pytest
```

**Using wildcard patterns to cache dependencies**
```yaml
steps:
Expand Down
34 changes: 28 additions & 6 deletions __tests__/cache-restore.test.ts
Expand Up @@ -10,6 +10,16 @@ describe('restore-cache', () => {
'd8110e0006d7fb5ee76365d565eef9d37df1d11598b912d3eb66d398d57a1121';
const requirementsLinuxHash =
'2d0ff7f46b0e120e3d3294db65768b474934242637b9899b873e6283dfd16d7c';
const poetryLockHash =
'571bf984f8d210e6a97f854e479fdd4a2b5af67b5fdac109ec337a0ea16e7836';
const poetryConfigOutput = `
cache-dir = "/Users/patrick/Library/Caches/pypoetry"
experimental.new-installer = false
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.path = "{cache-dir}/virtualenvs" # /Users/patrick/Library/Caches/pypoetry/virtualenvs
`;
patrick91 marked this conversation as resolved.
Show resolved Hide resolved

// core spy
let infoSpy: jest.SpyInstance;
Expand Down Expand Up @@ -47,6 +57,9 @@ describe('restore-cache', () => {
if (input.includes('pip')) {
return {stdout: 'pip', stderr: '', exitCode: 0};
}
if (input.includes('poetry')) {
return {stdout: poetryConfigOutput, stderr: '', exitCode: 0};
}

return {stdout: '', stderr: 'Error occured', exitCode: 2};
});
Expand Down Expand Up @@ -82,7 +95,8 @@ describe('restore-cache', () => {
],
['pip', '3.8.12', '__tests__/data/requirements.txt', requirementsHash],
['pipenv', '3.9.1', undefined, pipFileLockHash],
['pipenv', '3.9.12', '__tests__/data/requirements.txt', requirementsHash]
['pipenv', '3.9.12', '__tests__/data/requirements.txt', requirementsHash],
['poetry', '3.9.1', undefined, poetryLockHash]
])(
'restored dependencies for %s by primaryKey',
async (packageManager, pythonVersion, dependencyFile, fileHash) => {
Expand Down Expand Up @@ -138,7 +152,8 @@ describe('restore-cache', () => {
],
['pip', '3.8.12', '__tests__/data/requirements.txt', pipFileLockHash],
['pipenv', '3.9.1', undefined, requirementsHash],
['pipenv', '3.9.12', '__tests__/data/requirements.txt', requirementsHash]
['pipenv', '3.9.12', '__tests__/data/requirements.txt', requirementsHash],
['poetry', '3.9.1', undefined, requirementsHash]
])(
'restored dependencies for %s by primaryKey',
async (packageManager, pythonVersion, dependencyFile, fileHash) => {
Expand All @@ -154,10 +169,17 @@ describe('restore-cache', () => {
);
await cacheDistributor.restoreCache();
let result = '';
if (packageManager !== 'pipenv') {
result = `Cache restored from key: ${fileHash}`;
} else {
result = 'pipenv cache is not found';

switch (packageManager) {
case 'pip':
result = `Cache restored from key: ${fileHash}`;
break;
case 'pipenv':
result = 'pipenv cache is not found';
break;
case 'poetry':
result = 'poetry cache is not found';
break;
}

expect(infoSpy).toHaveBeenCalledWith(result);
Expand Down
44 changes: 44 additions & 0 deletions __tests__/cache-save.test.ts
Expand Up @@ -11,6 +11,8 @@ describe('run', () => {
'd8110e0006d7fb5ee76365d565eef9d37df1d11598b912d3eb66d398d57a1121';
const requirementsLinuxHash =
'2d0ff7f46b0e120e3d3294db65768b474934242637b9899b873e6283dfd16d7c';
const poetryLockHash =
'571bf984f8d210e6a97f854e479fdd4a2b5af67b5fdac109ec337a0ea16e7836';

// core spy
let infoSpy: jest.SpyInstance;
Expand Down Expand Up @@ -114,6 +116,22 @@ describe('run', () => {
);
expect(setFailedSpy).not.toHaveBeenCalled();
});

it('should not save cache for pipenv', async () => {
inputs['cache'] = 'pipenv';

await run();

expect(getInputSpy).toHaveBeenCalled();
expect(debugSpy).toHaveBeenCalledWith(
`paths for caching are ${__dirname}`
);
expect(getStateSpy).toHaveBeenCalledTimes(3);
expect(infoSpy).toHaveBeenCalledWith(
`Cache hit occurred on the primary key ${requirementsHash}, not saving cache.`
);
expect(setFailedSpy).not.toHaveBeenCalled();
});
});

describe('action saves the cache', () => {
Expand Down Expand Up @@ -168,6 +186,32 @@ describe('run', () => {
);
expect(setFailedSpy).not.toHaveBeenCalled();
});

it('saves cache from poetry', async () => {
inputs['cache'] = 'poetry';
getStateSpy.mockImplementation((name: string) => {
if (name === State.CACHE_MATCHED_KEY) {
return poetryLockHash;
} else if (name === State.CACHE_PATHS) {
return JSON.stringify([__dirname]);
} else {
return requirementsHash;
}
});

await run();

expect(getInputSpy).toHaveBeenCalled();
expect(getStateSpy).toHaveBeenCalledTimes(3);
expect(infoSpy).not.toHaveBeenCalledWith(
`Cache hit occurred on the primary key ${poetryLockHash}, not saving cache.`
);
expect(saveCacheSpy).toHaveBeenCalled();
expect(infoSpy).toHaveBeenLastCalledWith(
`Cache saved with the key: ${requirementsHash}`
);
expect(setFailedSpy).not.toHaveBeenCalled();
});
});

afterEach(() => {
Expand Down