Unable to build requirement with pip

[paolostancato]

Description:
(Possibly related to #824?)

Context: Runners are self hosted on kubernetes.
When installing packages with pip that require building, gcc is not able to find Python.h apparently because it's looking at the wrong path:

2024-04-04T03:00:10.1977421Z       gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.11.9/x64/include/python3.11 -c src/snappy/crc32c.c -o build/temp.linux-x86_64-cpython-311/src/snappy/crc32c.o
2024-04-04T03:00:10.1979214Z       gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.11.9/x64/include/python3.11 -c src/snappy/snappymodule.cc -o build/temp.linux-x86_64-cpython-311/src/snappy/snappymodule.o
2024-04-04T03:00:10.1980416Z       src/snappy/snappymodule.cc:29:10: fatal error: Python.h: No such file or directory
2024-04-04T03:00:10.1981026Z          29 | #include "Python.h"
2024-04-04T03:00:10.1981312Z             |          ^~~~~~~~~~
2024-04-04T03:00:10.1981584Z       compilation terminated.
2024-04-04T03:00:10.1981963Z       error: command '/usr/bin/gcc' failed with exit code 1

Include directory is set to

/opt/hostedtoolcache/Python/3.11.9/x64/include/python3.11

while Python.h resides in

/home/runner/_work/_tool/Python/3.11.9/x64/include/python3.11/Python.h

setup-python.log

Action version:
actions/setup-python@v5' (SHA:82c7e631bb3cdc910f68e0081d67478d79c6982d)

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:

Repro steps:

name: Debug

on:
  workflow_call:
  workflow_dispatch:

jobs:
  job:
    runs-on: ${{ vars.K8S_RUNNER }}
    strategy:
      fail-fast: false
    steps:
      - name: Install pre-requisits
        run: |
          sudo apt install -y libsnappy-dev

      - name: Set up Python 3.11
        uses: actions/setup-python@v5
        with:
          python-version: 3.11

      - name: Install requirements
        run: |
          env
          python -m pip install --upgrade pip pip-tools
          CPUCOUNT=1 pip install python-snappy==0.6.1

[HarithaVattikuti]

Hello @paolostancato
Thank you for creating this issue. We will investigate it and get back to you as soon as we have some feedback.

[aparnajyothi-y]

Hello @paolostancato, the gcc compiler is looking for the Python.h file in the directory specified by /opt/hostedtoolcache/Python/3.11.9/x64/include/python3.11, but the file is actually located in /home/runner/_work/_tool/Python/3.11.9/x64/include/python3.11.
This discrepancy could be due to a misconfiguration of the environment variables . The AGENT_TOOLSDIRECTORY and RUNNER_TOOL_CACHE should be set such that they point to the correct directories where the necessary tools and files are located.
The RUNNER_TOOL_CACHE is a predefined environment variable in GitHub Actions runners that directs to a directory for caching tools and dependencies, facilitating reuse across workflow runs. This is elaborated in the default environment variables documentation.

For self-hosted runners on Linux and Windows, the AGENT_TOOLSDIRECTORY environment variable allows the specification of a custom directory for tool installations and caching, as detailed in the advanced usage documentation.
When you designate a custom directory with AGENT_TOOLSDIRECTORY, the RUNNER_TOOL_CACHE no longer defaults to the runner's preset directory.

One of the possible workaround is mentioned below
In a self-hosted runner setup on Kubernetes, we might need to adjust the setup to ensure that these environment variables are correctly set. This could involve modifying the runner's configuration or adjusting the Kubernetes deployment to correctly mount the necessary directories.

spec:
template:
spec:
containers:
- name: github-runner
image: my-docker/github-actions-runner:latest
env:
- name: AGENT_TOOLSDIRECTORY
value: ""/path/to/tools""
- name: RUNNER_TOOL_CACHE
value: ""/path/to/cache""

[aparnajyothi-y]

Hello @paolostancato, Please confirm the above information resolved the issue.

[paolostancato]

Hi @aparnajyothi-y , thanks for your prompt response, I'll close the issue because I couldn't reproduce it again