Load GOOGLE_APPLICATION_CREDENTIALS json content via an environment variable instead of a file

[leighmcculloch]

👋

What from I can see the golang.org/x/oauth2/google package only supports loading the JSON application credentials file from the path given in the GOOGLE_APPLICATION_CREDENTIALS environment variable, without digging deep into the inner workings.

When deploying go apps on Heroku or Pivotal Web Services secrets are normally kept in environment variables. Code and files get pushed together on every deploy and it's not ideal to keep the credentials JSON with the code and it's not easy to manage the credentials file alongside the code since it needs to be committed in the case of Heroku, or present at every push in the case of Pivotal Web Services,

I figured out how to do this but I had to dig through the source and multiple docs. This was the resulting code:

json := os.Getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON") // `{"type": "service_account", "project_id": "my-project", ...}`
ctx := context.Background()
jwtConfig, err := google.JWTConfigFromJSON([]byte(json), datastore.ScopeDatastore)
if err != nil {
	...
}
ts := jwtConfig.TokenSource(ctx)
datastoreClient, err := datastore.NewClient(ctx, projectID, option.WithTokenSource(ts))

I think there need to be another environment variable that will pickup the config and set default credentials without needing it to be in a file, or we need clearer documentation on how to load credentials from elsewhere.

[zombiezen]

Hey Leigh. Application Default Credentials is a cross-language specification, so we would need to change how all languages work in order to support this use case. As you've seen, it is possible to pass along credentials in your own way, but it sounds like we could improve our documentation around this.

@rakyll, any thoughts on how we could improve?

[rakyll]

We are documenting how to construct clients with custom token sources (see https://github.com/googlecloudplatform/google-cloud-go#authorization), but we can improve the text around it and give a more comprehensive example rather than having the two line reference.

[broady]

Another pattern you might want to look at is to encrypt the key JSON and pass along the secrets in environment variables. You'd need to decrypt the file on application startup and set the GOOGLE_APPLICATION_CREDENTIALS env var.

[leighmcculloch]

@zombiezen: Documentation could solve this problem.

@rakyll: Thanks for the link. I was using the Google Developers page https://developers.google.com/identity/protocols/application-default-credentials below as reference point which hadn't mentioned it. The docs seem to live in few places and is confusing to navigate because of that. I see that I should have posted this issue in the googlecloudplatform/google-cloud-go project. The docs you linked to were clearer and it would be helpful of all the Google Developer docs were in sync.

@broady: I don't think I'd want to further complicate the solution by introducing encryption. If the encryption key lives in an environment variable the credentials may as well, in which case the solution I shared above has less moving parts.

---

I think clearer documentation for this alternative way of providing the credentials to the library would go a long way, but for devs using non-AppEngine/Compute services like Heroku and PWS there is still increased friction. It wouldn't surprise me if the friction of figuring out how to do auth within these PaaS providers contributes to lost adoption.

Closing since @rakyll pointed out this was opened in the wrong project.

[avsuresh04]

Hi @leighmcculloch,

Do you have the full code to set JSON content inline rather than file. I need in .NET but may be I'll convert from Java to .NET.

Thanks
Anil Jain

[leighmcculloch]

The JSON content isn't something that you create, but something that you get from the Google Cloud Console. You'll want to follow the instructions here to create a new service account key, which is a type of credential you can create which is attached to a service. When it gives you the option of key type, select JSON.

It'll end up looking something like:

{"type": "service_account", "project_id": "my-project", ...}

[avsuresh04]

Hi @leighmcculloch
I already have all the credentials required in the form of JSON. All I was looking was for something that could be loaded dynamically in json format pointed from environment variable rather than having it in a specific json file.

[Macilias]

why is this closed? I´m looking also for a way to bypass using dedicated file for credentials. Plenty environments allow only variables outside of the mounted environment.

[jba]

Have you tried CredentialsFromJSON?

[jeanbza]

Closing due to staleness. Please see CredentialsFromJSON, which should address this feature request.

[black-snow]

+1
For now I have to fetch the contents from ENV and write it to a temporary file in order to provide it to the gradle plugin (java project here). But the file contents may leak / the file might persist which is bad.

[broady]

Hi folks, an update on this feature request.

We won't be implementing this any time soon, but it's on our list of ideas for improvements to our auth libraries.

[fishhead108]

Any news here?

[edi]

Looking for the same, but in PHP ... I don't want my credentials stored on a GH Repo for automatic deployment.

As a workaround, during build time, I'm taking away the ENV var I've set, and creating a file called credentials.json under a folder with no public access to which the code points in PHP afterwads like so:

putenv('GOOGLE_APPLICATION_CREDENTIALS='. CACHE_PATH .'/credentials.json');

[codyoss]

There is no update to share as of right now, but I have raised this feature request internally. If we supported this we would support such an environment variable in all languages, not just Go.

[tbpg]

For Go in particular, you can use option.WithCredentialsJSON and combine it with os.Getenv to do this today.

Something like:

foo.NewClient(ctx, option.WithCredentialsJSON([]byte(os.Getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON"))))

If you want to make it work for local development and prod, you can check if the _JSON env var exists first, and fall back to no option.WithCredentialsJSON if not.

I don't know if there is a similar feature in PHP.

[Sytten]

I think this really important for dev environments. Currently the SDK doesn't encrypt credentials (weird decision in my opinion) so devs that care about security must store them somewhere else. What I would like is decrypt the file in memory and pass it to the application directly.

[treeder]

Just came across this searching for something else, but we do it like this:

1. Encode the credentials JSON:

base64 -w 0 account.json

Add that encoded string to your environment variables, eg:

G_KEY=${ENCODED_CREDENTIALS}

Then when your app starts up, decode it into a ClientOption:

creds := os.Getenv("G_KEY")
serviceAccountJSON, err := base64.StdEncoding.DecodeString(serviceAccountEncoded)
opts := []option.ClientOption{}
opts = append(opts, option.WithCredentialsJSON(serviceAccountJSON))

Then you use that opts with any Google Cloud client, eg:

firestore.NewClient(ctx, projectID, opts)

[hanikesn]

All these workarounds work well for your own applications, but the real challenge is working with 3rd-party tooling written in Go, which doesn't behave like tools written in other languages.

[treeder]

That works for all the official Firebase and Google Cloud Go libraries.

[hanikesn]

That works for all the official Firebase and Google Cloud Go libraries.

Only when you have access to the source code and are not using precompiled binaries.