Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Tool Path handling for self-hosted runners #466

Merged
merged 6 commits into from Jul 26, 2022

Conversation

techman83
Copy link
Contributor

@techman83 techman83 commented Jul 18, 2022

Description:
This removes the manual paths that were set to address issues with self-hosted runners. However upon further investigation my previous attempt wasn't quite correct. It did restore the out of the box functionality for self-hosted runners, but it didn't address what #394 was intending to fix.

An improvement could be to handle AGENT_TOOLSDIRECTORY further in the code, as it seems a bit incorrect to override the RUNNER_TOOL_CACHE, however this seems to address all use cases and is an improvement 🙂

Related issue:
#459
Check list:

  • Mark if documentation changes are required.
  • Mark if tests were added or updated to cover the changes.

@techman83 techman83 requested a review from a team July 18, 2022 15:22
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@IvanZosimov
Copy link
Contributor

Hi, @techman83 👋 Unfortunately, Python binaries for macOS are indeed non-relocatable (you can check it by yourself using a self-hosted runner with macOS and setup-python v4.0.0 without setting AGENT_TOOLSDIRECTORY) and there is no simple way to make them relocatable, so may I ask you to get back to previous logic of assigning AGENT_TOOLSDIRECTORY but only for macOS ( '/Users/runner/hostedtoolcache' ) ? In that case, we also have to revert some changes in README.md namely related to creating '/Users/runner/hostedtoolcache' on macOS and granting permissions.

@techman83
Copy link
Contributor Author

@IvanZosimov that's unfortunate, I don't have access to a Mac, but I will expand the PR to cover that case. I'll add some clear documentation to go with it.

@techman83
Copy link
Contributor Author

@IvanZosimov I have a way to validate the issue, and will have to document / ensure we complain loudly that AGENT_TOOLSDIRECTORY is not valid for mac, self hosted or otherwise. Though I may submit a PR upstream, going down the rabbit hole to understand why, I can see it.

@techman83
Copy link
Contributor Author

I have some small tweaks to the commits messages to do, and the docs to add. But that should be a clean and clear message in the process. Feedback welcome!

src/setup-python.ts Outdated Show resolved Hide resolved
@techman83
Copy link
Contributor Author

Alright, I tweaked it slightly as AGENT_TOOLSDIRECTORY isn't applicable for Mac due to the non-relocatable flag. It's probably worth raising a PR upstream, as it looks like a relatively minor change (at least at a glance). Plus fixed up the commits + conflicts.

@techman83
Copy link
Contributor Author

Noticed the prettier was upset, so ran that through and fixed it up.

@IvanZosimov
Copy link
Contributor

Hi, @techman83 👋 Thanks for the changes, we are going to merge this PR soon, could you, please, sync it with the main branch to resolve conflicts?

This updates and simplies the tool cache documentation to match the implementation in
both  and

  Relates actions#459
This fixes the tool cache path for self-hosted runners, along
with handling AGENT_TOOLSDIRECTORY for both hosted + self-hosted.

    Fixes actions#459
Shared libraries for the Mac python builds are not configured with the
relocatable flag, thus must always be configured with the hosted path.

Relates actions#459
Ensure that the path requirements and reasoning is clear, to reduce
confusion when using self-hosted, or attempting to set an
'AGENT_TOOLSDIRECTORY' environment variable.
@techman83
Copy link
Contributor Author

Hi @IvanZosimov, all done!

src/setup-python.ts Outdated Show resolved Hide resolved
@marko-zivic-93 marko-zivic-93 merged commit a93d541 into actions:main Jul 26, 2022
@techman83 techman83 deleted the fix/tool_path branch July 27, 2022 00:36
welpaolo pushed a commit to canonical/spark-k8s-toolkit-py that referenced this pull request Aug 14, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/setup-python](https://togithub.com/actions/setup-python) |
action | major | `v2.2.2` -> `v4.7.0` |
|
[canonical/data-platform-workflows](https://togithub.com/canonical/data-platform-workflows)
| action | major | `v2` -> `v4` |

---

### Release Notes

<details>
<summary>actions/setup-python (actions/setup-python)</summary>

###
[`v4.7.0`](https://togithub.com/actions/setup-python/releases/tag/v4.7.0)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.6.1...v4.7.0)

In scope of this release, the support for reading python version from
pyproject.toml was added
([actions/setup-python#669).

```yaml
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version-file: pyproject.toml
```

##### Besides, it includes such changes as:

- Bump tough-cookie and
[@&#8203;azure/ms-rest-js](https://togithub.com/azure/ms-rest-js) by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[actions/setup-python#697
- Bump semver from 7.3.8 to 7.5.2 by
[@&#8203;dependabot](https://togithub.com/dependabot) in
[actions/setup-python#692
- Fix typos found by codespell by
[@&#8203;DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos)
in
[actions/setup-python#650

#### New Contributors

- [@&#8203;dariocurr](https://togithub.com/dariocurr) made their first
contribution in
[actions/setup-python#669
-
[@&#8203;DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos)
made their first contribution in
[actions/setup-python#650

**Full Changelog**:
actions/setup-python@v4...v4.7.0

###
[`v4.6.1`](https://togithub.com/actions/setup-python/releases/tag/v4.6.1)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.6.0...v4.6.1)

#### What's Changed

- Fix `allow-prereleases` sample configuration by
[@&#8203;mayeut](https://togithub.com/mayeut) in
[actions/setup-python#615
- Fix a incorrect link advanced-usage.md by
[@&#8203;siyuan0322](https://togithub.com/siyuan0322) in
[actions/setup-python#657
- Remove implicit dependency by
[@&#8203;nikolai-laevskii](https://togithub.com/nikolai-laevskii) in
[actions/setup-python#668
- Automatic update of configuration files from 05/23/2023 by
[@&#8203;github-actions](https://togithub.com/github-actions) in
[actions/setup-python#671
- Add warning for python 2.7 on release/v4 by
[@&#8203;dmitry-shibanov](https://togithub.com/dmitry-shibanov) in
[actions/setup-python#673

#### New Contributors

- [@&#8203;siyuan0322](https://togithub.com/siyuan0322) made their first
contribution in
[actions/setup-python#657
- [@&#8203;nikolai-laevskii](https://togithub.com/nikolai-laevskii) made
their first contribution in
[actions/setup-python#668

**Full Changelog**:
actions/setup-python@v4...v4.6.1

###
[`v4.6.0`](https://togithub.com/actions/setup-python/releases/tag/v4.6.0):
Add allow-prereleases input

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.5.0...v4.6.0)

In scope of this release we added a new input (`allow-prereleases`) to
allow [falling back to pre-release versions of Python when a matching GA
version of Python is not
available](https://togithub.com/actions/setup-python/pull/414)

```yaml
steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
    with:
      python-version: 3.12
      allow-prereleases: true
```

Besides, we added such changes as:

- Fix bug to trim new line for PyPy version:
[actions/setup-python#610
- Added pip dependency file to generate hash from it:
[actions/setup-python#604
- Improved error handling for saving and restoring cache:
[actions/setup-python#618
- Add warning if cache paths are empty:
[actions/setup-python#642

###
[`v4.5.0`](https://togithub.com/actions/setup-python/releases/tag/v4.5.0):
Fix cache issue for Poetry projects located in subfolders

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.4.0...v4.5.0)

In scope of this release we fixed cache issue for Poetry projects
located in subfolders
([actions/setup-python#446).
Besides that we updated json5 version from 2.2.0 to 2.2.3
([actions/setup-python#579).

###
[`v4.4.0`](https://togithub.com/actions/setup-python/releases/tag/v4.4.0):
Add support to install multiple python versions

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.3.1...v4.4.0)

In scope of this release we added support to [install multiple python
versions](https://togithub.com/actions/setup-python/pull/567). For this
you can try to use this snippet:

```yaml
    - uses: actions/setup-python@v4
      with:
        python-version: |
            3.8
            3.9
            3.10
```

Besides, we changed logic with throwing the error for GHES if cache is
unavailable to warn
([actions/setup-python#566).

###
[`v4.3.1`](https://togithub.com/actions/setup-python/releases/tag/v4.3.1):
Improve error handling and messages

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.3.0...v4.3.1)

In scope of this release we added improved error message to put
operating system and its version in the logs
([actions/setup-python#559).
Besides, the release

- fixes issue about [specifying architecture for pypy-nightly on
Windows](https://togithub.com/actions/setup-python/issues/518) with
related [pull
request](https://togithub.com/actions/setup-python/pull/520).
- improves error handling for Http Errors
([actions/setup-python#511).
- updates minimatch
([actions/setup-python#558).

###
[`v4.3.0`](https://togithub.com/actions/setup-python/releases/tag/v4.3.0)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.2.0...v4.3.0)

- Update [@&#8203;actions/core](https://togithub.com/actions/core) to
1.10.0 version
[#&#8203;517](https://togithub.com/actions/setup-python/issues/517)
- Update [@&#8203;actions/cache](https://togithub.com/actions/cache) to
3.0.4 version
[#&#8203;499](https://togithub.com/actions/setup-python/issues/499)
- Only use github.token on github.com
[#&#8203;443](https://togithub.com/actions/setup-python/issues/443)
- Improvement of documentation
[#&#8203;477](https://togithub.com/actions/setup-python/issues/477)
[#&#8203;479](https://togithub.com/actions/setup-python/issues/479)
[#&#8203;491](https://togithub.com/actions/setup-python/issues/491)
[#&#8203;492](https://togithub.com/actions/setup-python/issues/492)

###
[`v4.2.0`](https://togithub.com/actions/setup-python/releases/tag/v4.2.0):
Add check-latest input and bug fixes

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.1.0...v4.2.0)

In scope of this release we add
the [check-latest](https://togithub.com/actions/setup-python/pull/406) input.
If check-latest is set to true, the action first checks if the cached
version is the latest one. If the locally cached version is not the most
up-to-date, the version will then be downloaded
from [python-versions](https://togithub.com/actions/python-versions) repository.
By default check-latest is set to false. For PyPy it will to try to
reach https://downloads.python.org/pypy/versions.json

##### Example of usage:

```yaml
steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
    with:
      python-version: '3.9'
      check-latest: true
  - run: python --version
```

Besides, it includes such changes as

- Resolved logs for python-version and file inputs:
[actions/setup-python#465
- Added linux os release info to primary key:
[actions/setup-python#467
- Added fix to change Python versions for poetry:
[actions/setup-python#445
- Fix Tool Path handling for self-hosted runners:
[actions/setup-python#466

###
[`v4.1.0`](https://togithub.com/actions/setup-python/releases/tag/v4.1.0)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v4.0.0...v4.1.0)

In scope of this pull request we updated `actions/cache` package as the
new version contains fixes for [caching error
handling](https://togithub.com/actions/setup-python/pull/448). Moreover,
we added a new input
[update-environment](https://togithub.com/actions/setup-python/pull/411).
This option allows to specify if the action shall update environment
variables (default) or not.

##### Update-environment input

```yaml
    - name: setup-python 3.9
      uses: actions/setup-python@v4
      with:
        python-version: 3.9
        update-environment: false
```

**Besides, we added such changes as:**

- Allow python-version-file to be a relative path:
[actions/setup-python#431
- Added new environment variables for `Cmake`:
[actions/setup-python#440
- Updated error message for resolveVersion:
[actions/setup-python#450
- Assign default value of `AGENT_TOOLSDIRECTORY` if not set:
[actions/setup-python#394

###
[`v4.0.0`](https://togithub.com/actions/setup-python/releases/tag/v4.0.0)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v3.1.4...v4.0.0)

##### What's Changed

- Support for `python-version-file` input:
[#&#8203;336](https://togithub.com/actions/setup-python/issues/336)

Example of usage:

```yaml
- uses: actions/setup-python@v4
  with:
    python-version-file: '.python-version' # Read python version from a file
- run: python my_script.py
```

There is no default python version for this `setup-python` major
version, the action requires to specify either `python-version` input or
`python-version-file` input. If the `python-version` input is not
specified the action will try to read required version from file from
`python-version-file` input.

- Use pypyX.Y for PyPy `python-version` input:
[#&#8203;349](https://togithub.com/actions/setup-python/issues/349)

Example of usage:

```yaml
- uses: actions/setup-python@v4
  with:
    python-version: 'pypy3.9' # pypy-X.Y kept for backward compatibility
- run: python my_script.py
```

- `RUNNER_TOOL_CACHE` environment variable is equal
`AGENT_TOOLSDIRECTORY`:
[#&#8203;338](https://togithub.com/actions/setup-python/issues/338)

- Bugfix: create missing `pypyX.Y` symlinks:
[#&#8203;347](https://togithub.com/actions/setup-python/issues/347)

- `PKG_CONFIG_PATH` environment variable:
[#&#8203;400](https://togithub.com/actions/setup-python/issues/400)

- Added `python-path` output:
[#&#8203;405](https://togithub.com/actions/setup-python/issues/405)
    `python-path` output contains Python executable path.

- Updated `zeit/ncc` to `vercel/ncc` package:
[#&#8203;393](https://togithub.com/actions/setup-python/issues/393)

- Bugfix: fixed output for prerelease version of poetry:
[#&#8203;409](https://togithub.com/actions/setup-python/issues/409)

- Made `pythonLocation` environment variable consistent for Python and
PyPy:
[#&#8203;418](https://togithub.com/actions/setup-python/issues/418)

- Bugfix for `3.x-dev` syntax:
[#&#8203;417](https://togithub.com/actions/setup-python/issues/417)

- Other improvements:
[#&#8203;318](https://togithub.com/actions/setup-python/issues/318)
[#&#8203;396](https://togithub.com/actions/setup-python/issues/396)
[#&#8203;384](https://togithub.com/actions/setup-python/issues/384)
[#&#8203;387](https://togithub.com/actions/setup-python/issues/387)
[#&#8203;388](https://togithub.com/actions/setup-python/issues/388)

###
[`v3.1.4`](https://togithub.com/actions/setup-python/releases/tag/v3.1.4)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v3.1.3...v3.1.4)

#### What's Changed

In the scope of this patch release, the warning for deprecating Python
2.x was added in
[actions/setup-python#674
by [@&#8203;dmitry-shibanov](https://togithub.com/dmitry-shibanov)

For more information, check out
[actions/setup-python#672

**Full Changelog**:
actions/setup-python@v3...v3.1.4

###
[`v3.1.3`](https://togithub.com/actions/setup-python/releases/tag/v3.1.3):
Update actions/core to 1.10.0 for v3

[Compare
Source](https://togithub.com/actions/setup-python/compare/v3.1.2...v3.1.3)

In scope of this release we update actions/core to 1.10.0 for v3 major
tag:
[actions/setup-python#624

###
[`v3.1.2`](https://togithub.com/actions/setup-python/releases/tag/v3.1.2):
Update actions/cache version to 2.0.2

[Compare
Source](https://togithub.com/actions/setup-python/compare/v3.1.1...v3.1.2)

In scope of this release we updated `actions/cache` package as the new
version contains fixes related to GHES 3.5
([actions/setup-python#382)

###
[`v3.1.1`](https://togithub.com/actions/setup-python/releases/tag/v3.1.1):
Add &quot;cache-hit&quot; output and fix &quot;python-version&quot;
output for PyPy

[Compare
Source](https://togithub.com/actions/setup-python/compare/v3.1.0...v3.1.1)

This release introduces new output cache-hit
([actions/setup-python#373)
and fix python-version output for PyPy
([actions/setup-python#365)

The cache-hit output contains boolean value indicating that an exact
match was found for the key. It shows that the action uses already
existing cache or not. The output is available only if cache is enabled.

The python-version contains version of Python or PyPy.

###
[`v3.1.0`](https://togithub.com/actions/setup-python/releases/tag/v3.1.0):
Support caching poetry dependencies and caching on GHES 3.5

[Compare
Source](https://togithub.com/actions/setup-python/compare/v3.0.0...v3.1.0)

- In the scope of this release, we added [support for caching from GHES
3.5](https://togithub.com/actions/setup-python/issues/362) and fixed the
download issue for files > 2GB during restore.
-   Caching poetry dependencies

```yaml
steps:
- uses: actions/checkout@v3
- name: Install poetry
  run: pipx install poetry
- uses: actions/setup-python@v3
  with:
    python-version: '3.9'
    cache: 'poetry'
- run: poetry install
- run: poetry run pytest
```

###
[`v3.0.0`](https://togithub.com/actions/setup-python/releases/tag/v3.0.0)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v2.3.4...v3.0.0)

#### What's Changed

- Update default runtime to node16
([actions/setup-python#340)
- Update `package-lock.json` file version to 2, `@types/node` to
16.11.25 and `typescript` to 4.2.3
([actions/setup-python#341)
- Remove legacy `pypy2` and `pypy3` keywords
([actions/setup-python#342)

##### Breaking Changes

With the update to Node 16, all scripts will now be run with Node 16
rather than Node 12.

This new major release removes support of legacy `pypy2` and `pypy3`
keywords. Please use more specific and flexible syntax to specify a PyPy
version:

```yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version:
        - 'pypy-2.7' # the latest available version of PyPy that supports Python 2.7
        - 'pypy-3.8' # the latest available version of PyPy that supports Python 3.8
        - 'pypy-3.8-v7.3.8' # Python 3.8 and PyPy 7.3.8
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v3
      with:
        python-version: ${{ matrix.python-version }}
```

See more usage examples in the
[documentation](https://togithub.com/actions/setup-python#specifying-a-pypy-version)

###
[`v2.3.4`](https://togithub.com/actions/setup-python/releases/tag/v2.3.4)

[Compare
Source](https://togithub.com/actions/setup-python/compare/v2.3.3...v2.3.4)

#### What's Changed

In the scope of this patch release, the warning for deprecating Python
2.x was added in
[actions/setup-python#675
by [@&#8203;dmitry-shibanov](https://togithub.com/dmitry-shibanov)

For more information, check out
[actions/setup-python#672

**Full Changelog**:
actions/setup-python@v2...v2.3.4

###
[`v2.3.3`](https://togithub.com/actions/setup-python/releases/tag/v2.3.3):
Update actions/core to 1.10.0 for v2

[Compare
Source](https://togithub.com/actions/setup-python/compare/v2.3.2...v2.3.3)

In scope of this release we update actions/core to 1.10.0 for v2 major
tag:
[actions/setup-python#533.

###
[`v2.3.2`](https://togithub.com/actions/setup-python/releases/tag/v2.3.2):
Update primary and restore keys for pip

[Compare
Source](https://togithub.com/actions/setup-python/compare/v2.3.1...v2.3.2)

In scope of this release we [include a version of python in restore and
primary cache keys for
pip](https://togithub.com/actions/setup-python/pull/303). Besides, we
add temporary fix for Windows caching
[issue](https://togithub.com/actions/setup-python/pull/332), that the
`pip cache dir` command returns non zero exit code or writes to stderr.
Moreover we updated [node-fetch
dependency](https://togithub.com/actions/setup-python/pull/327).

###
[`v2.3.1`](https://togithub.com/actions/setup-python/releases/tag/v2.3.1):
Update actions/cache version to 1.0.8

[Compare
Source](https://togithub.com/actions/setup-python/compare/v2.3.0...v2.3.1)

We have
updated [actions/cache](https://togithub.com/actions/toolkit/blob/main/packages/cache/RELEASES.md#108) dependency
version to 1.0.8 to support 10GB cache upload

###
[`v2.3.0`](https://togithub.com/actions/setup-python/releases/tag/v2.3.0):
Support caching dependencies

[Compare
Source](https://togithub.com/actions/setup-python/compare/v2.2.2...v2.3.0)

This release introduces dependency caching support
([actions/setup-python#266)

#### Caching dependencies.

The action has a built-in functionality for caching and restoring
pip/pipenv dependencies. The `cache` input is optional, and caching is
turned off by default.

Besides, this release introduces dependency caching support for mono
repos and repositories with complex structure.

By default, the action searches for the dependency file
(requirements.txt for pip or Pipfile.lock for pipenv) in the whole
repository. Use the `cache-dependency-path` input for cases when you
want to override current behaviour and use different file for hash
generation (for example requirements-dev.txt). This input supports
wildcards or a list of file names for caching multiple dependencies.

##### Caching pip dependencies:

    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: '3.9'
        cache: 'pip'
    - run: pip install -r requirements.txt
    - run: pip test

##### Caching pipenv dependencies:

    steps:
    - uses: actions/checkout@v2
    - name: Install pipenv
      run: pipx install pipenv
    - uses: actions/setup-python@v2
      with:
        python-version: '3.9'
        cache: 'pipenv'
    - run: pipenv install
    - run: pipenv test

##### Change dependency file:

    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: '3.9'
        cache: 'pip'
        cache-dependency-path: '**/requirements-dev.txt'
    - run: pip install -r subdirectory/requirements-dev.txt
    - run: pip test

</details>

<details>
<summary>canonical/data-platform-workflows
(canonical/data-platform-workflows)</summary>

###
[`v4`](https://togithub.com/canonical/data-platform-workflows/compare/v3...v4)

[Compare
Source](https://togithub.com/canonical/data-platform-workflows/compare/v3...v4)

###
[`v3`](https://togithub.com/canonical/data-platform-workflows/compare/v2...v3)

[Compare
Source](https://togithub.com/canonical/data-platform-workflows/compare/v2...v3)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 1am and before 2am on saturday"
in timezone Etc/UTC, Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about these
updates again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/canonical/spark-k8s-toolkit-py).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi40MC4zIiwidXBkYXRlZEluVmVyIjoiMzYuNDAuMyIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants