gsutil isn't authenticated

[pvanderlinden]

TL;DR

gsutil isn't authenticated even though the docs mention it is possible

Expected behavior

gsutil is authenticated

Observed behavior

I get the error "401 Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket."

Action YAML

name: Docs build & upload

on:
  push:
    branches:
      - main

env:
  GCS_DEST: gs://...

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11"]
    steps:
      - uses: actions/checkout@v4
      - name: "google cloud auth"
        uses: "google-github-actions/auth@v2"
        with:
          credentials_json: "${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}"
      - name: publish
        if: ${{ github.ref == 'refs/heads/main' }}
        run: |
          gsutil -m rsync -x 'compiled/.*$|.*\.msgpack$' -R -r target  ${{ env.GCS_DEST }}

Log output

No response

Additional information

No response

[github-actions]

Hi there @pvanderlinden 👋!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

[sethvargo]

Hi @pvanderlinden - I'm not seeing where you're using gsutil in your example, did you mean gcloud? You need to use google-github-actions/setup-gcloud@v2 to install and configure gcloud.

[pvanderlinden]

Hi @pvanderlinden - I'm not seeing where you're using gsutil in your example, did you mean gcloud? You need to use google-github-actions/setup-gcloud@v2 to install and configure gcloud.

Sorry, apparently I posted the version with the work around: using gcloud storage instead of gsutil works. I adjusted the example, as I tried it originally. gcloud and gsutil already are pre installed on the github actions image. Is gcloud storage going to replace gsutil? If so, it might be good to mention in the docs someone should use gcloud storage instead of gsutil as the second doesn't work (while it currently says it does).

[sethvargo]

Hi @pvanderlinden, as noted in the README prerequisites, you must use gcloud/gsutil version 390.0.0 or later. We don't control which version of gcloud/gsutil is pre-installed on the GitHub Actions managed runners, and it may be older than the supported version.

[pvanderlinden]

According to the documentation it's uptodate enough. https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2204-Readme.md#cli-tools

I was going to double check, but github has some issues at the moment, and the workflow doesn't run.

[pvanderlinden]

Hi @pvanderlinden, as noted in the README prerequisites, you must use gcloud/gsutil version 390.0.0 or later. We don't control which version of gcloud/gsutil is pre-installed on the GitHub Actions managed runners, and it may be older than the supported version.

I just double checked. This is the result of running the command in a github actions workflow:

> gcloud --version
  
Google Cloud SDK 469.0.0
alpha 2024.03.15
beta 2024.03.15
bq 2.1.1
bundled-python3-unix 3.11.8
core 2024.03.15
gcloud-crc32c 1.0.0
gsutil 5.27

[sethvargo]

You still need actions/setup-gcloud (you can set install: false to bypass installation), because something has to run gcloud auth login...

[pvanderlinden]

You still need actions/setup-gcloud (you can set install: false to bypass installation), because something has to run gcloud auth login...

Isn't this what the "auth" action is supposed to do? It authenticates gcloud utilities (except gsutil/bq apparently)

      - name: "google cloud auth"
        uses: "google-github-actions/auth@v2"
        with:
          credentials_json: "${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}"

[sethvargo]

The auth action creates credentials files and exports environment variables. Unfortunately gcloud (and gsutil and bq) do not read any environment variables to source their authentication. auth can't run the commands to authenticate those tools because it doesn't install them on the system (and can't guarantee they are installed (e.g. self-hosted runners)).

[pvanderlinden]

The auth action creates credentials files and exports environment variables. Unfortunately gcloud (and gsutil and bq) do not read any environment variables to source their authentication. auth can't run the commands to authenticate those tools because it doesn't install them on the system (and can't guarantee they are installed (e.g. self-hosted runners)).

Thanks. This explains the issue. However, the current auth action actually authenticates the cli tool gcloud (as otherwise my work around wouldn't work) , but not gsutil and bq. This makes this action confusing, especially with the documentation saying "To use the bq or gsutil tools, use the Google Cloud SDK version 390.0.0 or newer.". I think it would enhance the documentation by a lot if it is actually mentioned in the documentation you need the setup actions as well to use these tools.

[sethvargo]

auth does not authenticate gcloud. setup-gcloud authenticates gcloud with the credentials from auth.

[sethvargo]

Hi @pvanderlinden does that answer your question?