GSClient auth fails with token-based application credentials.json

[joconnor-ecaa]

The following

gcloud auth application-default login
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/application_default_credentials.json
python -c "from cloudpathlib import GSPath; GSPath('gs://private_file').download_to('.')"

fails with

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/joe/cloudpathlib/cloudpathlib/cloudpath.py", line 171, in __call__
    cls.__init__(new_obj, cloud_path, *args, **kwargs)  # type: ignore[type-var]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/cloudpathlib/cloudpath.py", line 230, in __init__
    client = self._cloud_meta.client_class.get_default_client()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/cloudpathlib/client.py", line 104, in get_default_client
    cls._default_client = cls()
                          ^^^^^
  File "/Users/joe/cloudpathlib/cloudpathlib/gs/gsclient.py", line 88, in __init__
    self.client = StorageClient.from_service_account_json(application_credentials)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 109, in from_service_account_json
    return cls.from_service_account_info(credentials_info, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 76, in from_service_account_info
    credentials = service_account.Credentials.from_service_account_info(info)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/oauth2/service_account.py", line 240, in from_service_account_info
    signer = _service_account_info.from_dict(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/auth/_service_account_info.py", line 50, in from_dict
    raise exceptions.MalformedError(
google.auth.exceptions.MalformedError: Service account info was not in the expected format, missing fields token_uri, client_email.

This is because the logic in the GSClient init assumes that if a GOOGLE_APPLICATION_CREDENTIALS file exists, it is in the format of a service account JSON key (i.e. the call to from_service_account_json).

When using workload identity federation GOOGLE_APPLICATION_CREDENTIALS is in a different format (see here).

It is possible to work around this with existing functionality, e.g. explicitly creating a google.storage.client.Client or credentials object. However it would be nice if GSClient and GSPath "just work" with workload identity federation. I've been monkeypatching GSClient and GSPath to achieve this in a few projects.

The simplest workaround is probably to replace the call to from_service_account_json with a call to google.auth.load_credentials_from_file.

[pjbull]

Thanks for the detailed report. From what I read, looks like this wouldn't be a breaking change for users that rely on the way things work now, so happy to take the fix you suggested.

Do you know if there is a good way to get and test the workload identity credentials?

[joconnor-ecaa]

Do you know if there is a good way to get and test the workload identity credentials?

You can set up workload identity federation by following this guide:

https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication-from-github-actions

Adding a CI workflow that authenticates using the github action shown in that blog and then runs the live GCS tests should do the trick. Unsure if there's a simpler way to test.