cant use credentials_json after an hour and half

[ShaiShalevSQream]

TL;DR

After about hour and half, using credentials_json is not effective. in the same workflow job.
I understand using token has timelimit, but as i understand credentials_json should not have timelimit.

Expected behavior

Be able to preform actions as "kubectl get pods" using credentials_json even after long period of time in the same job.

Observed behavior

i have a workflow with one job. and few steps.
everything working fine, if for example the steps of watch logs take 20-30 minutes.
but if it takes hour and half for example,

then the last step "get pods" gets error of
error: You must be logged in to the server (Unauthorized)
and the step remove chart gets error of
Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

Action YAML

jobs:
  tier1:
    if: ${{ github.event_name == 'workflow_dispatch' }}
    strategy:
      fail-fast: false
      matrix:
        suite: ${{ fromJSON(github.event.inputs.suite) }}
    name: ...
    runs-on: [ gcp-self-hosted-org ]
    outputs:
      kubectl_config: ${{ steps.kubectl_secret.outputs.kubectl_config }}
    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      - name: Checkout devops-scripts code
        uses: actions/checkout@v4

      - name: update environment parameters
        run: |
          ...

      - name: authenticate to google
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_CI_CD_SECRET }}

      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v2
        with:
          project_id: ${{ env.project }}

      - name: get credentials for the kubectl
        id: kube_cred
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ env.cluster_name }}
          location: ${{ env.region }}
          project_id: ${{ env.project }}

      - name: add helm repository
        run: |
          helm repo add ....
          helm repo update
          helm upgrade -i ${{ env.RELEASE_NAME }} ....

      - name: watch logs
        run: |
          
          kubectl get pods -n default
          kubectl wait --for=jsonpath='{.status.phase}'=Running pod/${{ env.POD_NAME }} -n default --timeout=120s
          kubectl logs -n default ${{ env.POD_NAME }} -f | tee ${{ env.ARTIFACT_NAME }}.txt

      - name: get pods
        run: |
          kubectl get pods -n default

      - name: remove chart
        if: ${{always()}}
        run: |
          set +e
          helm uninstall ${{ env.RELEASE_NAME }} -n default

Log output

EXIT_CODE=$(kubectl get pods .. -n default -o json | jq -r '.status.containerStatuses[].state.terminated.exitCode')
  echo $EXIT_CODE
  if [ "$EXIT_CODE" != 0 ]; then
    echo "Failure"
    exit 1
  fi
  shell: /usr/bin/bash -e ***0***
  env:
    environment: ...
    connector: ...
    suite: [ 'blue' ]
    x_flag: false
    tenant: ..
    region: ...
    cloud_provider: ..
    cluster_name: ...
    project: ...
    domain_name: ...
    CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /home/runner/_work/tier1_shai/tier1_shai/gha-creds-eb4dc72a558f8d5c.json
    GOOGLE_APPLICATION_CREDENTIALS: /home/runner/_work/tier1_shai/tier1_shai/gha-creds-eb4dc72a558f8d5c.json
    GOOGLE_GHA_CREDS_PATH: /home/runner/_work/tier1_shai/tier1_shai/gha-creds-eb4dc72a558f8d5c.json
    CLOUDSDK_CORE_PROJECT: sqreamtech
    CLOUDSDK_PROJECT: sqreamtech
    GCLOUD_PROJECT: sqreamtech
    GCP_PROJECT: sqreamtech
    GOOGLE_CLOUD_PROJECT: sqreamtech
    CLOUDSDK_METRICS_ENVIRONMENT: github-actions-setup-gcloud
    CLOUDSDK_METRICS_ENVIRONMENT_VERSION: 2.1.0
    KUBECONFIG: /home/runner/_work/tier1_shai/tier1_shai/gha-kubeconfig-79fec1d14a0268a0
    KUBE_CONFIG_PATH: /home/runner/_work/tier1_shai/tier1_shai/gha-kubeconfig-79fec1d14a0268a0
    TESTING_FRAMEWORK_IMAGE: 89148da185
    POD_NAME: ...
    RELEASE_NAME: ...
    ARTIFACT_NAME: ...
error: You must be logged in to the server (Unauthorized)

Failure
Error: Process completed with exit code 1.

Additional information

My runner are self hosted runners. using runner-scale-set.

[github-actions]

Hi there @ShaiShalevSQream 👋!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

[sethvargo]

Hi @ShaiShalevSQream - this is an issue with get-gke-credentials, not auth. If you don't use the auth plugin (use_auth_provider), then get-gke-credentials generates an OAuth access token and puts it in the Kubeconfig.

You can either refresh auth before running kubectl get po (by using the get-gke-credentials action again) or install and use the auth plugin which handles refreshes.

[ShaiShalevSQream]

Thank you very much for your answer.