GCS - signed URLs (local development) #6268
Comments
|
To be honest for now the documentation should really reference: https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication |
|
@jpaulgs Thank you for this valuable feedback. We'll see what we can do to improve this. Much appreciated! I'll try to reproduce your code example as soon as I can. |
quartzmo
added
api: storage
|
@quartzmo Having the same issue in GKE with workload identity feature |
|
@frankyn Any idea about using Signed URL V2 in GKE with Workload Identity? |
|
Hi @quartzmo, it would need to follow a similar solution for Compute Engine instances IIRC. I thought the Ruby library support IAM SignBlob for Signed URLs* |
|
@quartzmo @frankyn response = service.sign_service_account_blob(name, request_body)
|
|
Implemented same in google cloud shell
Followed link: https://cloud.google.com/iam/docs/reference/rest/v1/projects.serviceAccounts/signBlob So i am curious what went wrong, can we use signBlob with workload identity |
|
@jpaulgs wrote:
Regarding this error, please see the answer provided by @dazuma on this Stack Overflow question:
|
|
@jpaulgs asked:
I will open a PR to improve the error detail messages for all |
|
Here is the new error message: |
|
@chvreddy I do not believe that you can use Workload Identity with the current signed URLs feature. If you find a way to do it, please post the solution here. Thank you! |
|
Hi folks, reopening. I missed follow-ups on this thread, apologies. There are two issues:
I provided a prototype that I used to make a successful request within a GKE cluster using Work Identity. I'm new to this environment but hopefully this can help move the conversation forward. @quartzmo I created a class IAMSigner to get past using an RSA object, and it would be helpful to have a way to provide a lambda that's called instead of the RSA sign method. require 'googleauth'
require 'google/cloud/storage'
require 'google/apis/iamcredentials_v1'
IAMCredentials = Google::Apis::IamcredentialsV1 # Alias the module
# The following is a hack to introduce IAMCredentialsService into the Storage library
# without modifying the existing library as a proof of concept.
class IAMSigner
def initialize(issuer)
@iam_credentials_client = IAMCredentials::IAMCredentialsService.new
# Get the environment configured authorization
scopes = ['https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/compute']
@iam_credentials_client.authorization = Google::Auth.get_application_default(scopes)
@issuer = issuer
end
def sign(digest, string_to_sign)
# Ignore digest
request = {
"payload": string_to_sign,
}
response = @iam_credentials_client.sign_service_account_blob(
"projects/-/serviceAccounts/#{@issuer}",
request,
{}
)
response.signed_blob
end
end
storage = Google::Cloud::Storage.new
storage_expiry_time = 5 * 60 # 5 minutes
bucket_name = "anima-frank"
file_name = "unnamed.jpg"
issuer = "workload-service-account@project-id.iam.gserviceaccount.com"
signing_key = IAMSigner.new issuer
url = storage.signed_url bucket_name, file_name, issuer: issuer,
signing_key: signing_key,
method: "GET", expires: storage_expiry_time,
version: :v4
puts urlAfter potential fixes, I'm thinking about something similar to the following: require 'googleauth'
require 'google/cloud/storage'
require 'google/apis/iamcredentials_v1'
IAMCredentials = Google::Apis::IamcredentialsV1 # Alias the module
issuer_and_signer = lambda do |string_to_sign|
iam_credentials_client = IAMCredentials::IAMCredentialsService.new
# Get the environment configured authorization
scopes = ['https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/compute']
iam_credentials_client.authorization = Google::Auth.get_application_default(scopes)
issuer = iam_credentials_client.authorization.issuer
request = {
"payload": string_to_sign,
}
response = @iam_credentials_client.sign_service_account_blob(
"projects/-/serviceAccounts/#{@issuer}",
request,
{}
)
[issuer, response.signed_blob]
end
storage = Google::Cloud::Storage.new
storage_expiry_time = 5 * 60 # 5 minutes
bucket_name = "anima-frank"
file_name = "unnamed.jpg"
url = storage.signed_url bucket_name, file_name, issuer_and_signer: issuer_and_signer
method: "GET", expires: storage_expiry_time,
version: :v4
puts url |
|
In the last example, should it be |
|
Completed example doesn't consider issuer being needed to construct string_to_sign. Let's follow Go's interface a little bit instead with googleapis/google-cloud-go#1130 (comment) So updated the example, it would be: ### PROTOTYPE CODE AND DOES NOT WORK!!!! #####
require 'googleauth'
require 'google/cloud/storage'
require 'google/apis/iamcredentials_v1'
IAMCredentials = Google::Apis::IamcredentialsV1 # Alias the module
iam_credentials_client = IAMCredentials::IAMCredentialsService.new
# Get the environment configured authorization
scopes = ['https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/compute']
iam_credentials_client.authorization = Google::Auth.get_application_default(scopes)
issuer = iam_credentials_client.authorization.issuer
signer = lambda do |string_to_sign|
request = {
"payload": string_to_sign,
}
response = iam_credentials_client.sign_service_account_blob(
"projects/-/serviceAccounts/#{issuer}",
request,
{}
)
response.signed_blob
end
storage = Google::Cloud::Storage.new
storage_expiry_time = 5 * 60 # 5 minutes
bucket_name = "anima-frank"
file_name = "unnamed.jpg"
url = storage.signed_url bucket_name, file_name, issuer: issuer, signer: signer
method: "GET", expires: storage_expiry_time,
version: :v4
puts url |
|
So the only public API or behavior change needed in this last example is that |
|
That's correct @quartzmo, I think maybe a new alias such as |
|
@chvreddy could you confirm if #6268 (comment) would help in this case? |
|
Thanks @frankyn let me apply this patch and see if we are able to create signed URL's |
|
Thanks for confirming @chvreddy. require 'googleauth'
require 'google/cloud/storage'
require 'google/apis/iamcredentials_v1'
IAMCredentials = Google::Apis::IamcredentialsV1 # Alias the module
# The following is a hack to introduce IAMCredentialsService into the Storage library
# without modifying the existing library as a proof of concept.
class IAMSigner
def initialize(issuer)
@iam_credentials_client = IAMCredentials::IAMCredentialsService.new
# Get the environment configured authorization
scopes = ['https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/compute']
@iam_credentials_client.authorization = Google::Auth.get_application_default(scopes)
@issuer = issuer
end
def sign(digest, string_to_sign)
# Ignore digest
request = {
"payload": string_to_sign,
}
response = @iam_credentials_client.sign_service_account_blob(
"projects/-/serviceAccounts/#{@issuer}",
request,
{}
)
response.signed_blob
end
end
storage = Google::Cloud::Storage.new
storage_expiry_time = 5 * 60 # 5 minutes
bucket_name = "anima-frank"
file_name = "unnamed.jpg"
issuer = "workload-service-account@project-id.iam.gserviceaccount.com"
signing_key = IAMSigner.new issuer
url = storage.signed_url bucket_name, file_name, issuer: issuer,
signing_key: signing_key,
method: "GET", expires: storage_expiry_time,
version: :v4
puts url |
sure @frankyn will apply this patch and see how is it going. Thanks! |
|
Update add in #7091 is pending release of google-cloud-storage gem. reopening until gem is released. |
|
@frankyn 🆒 Are you able to comment when the next version of the gem with the fix will be released? |
|
Hi @joedicator, I'm pending input from @quartzmo right now. Open PR with next version is #6993 |
|
Woot! Thanks @quartzmo! google-cloud-storage 1.27.0 released Closing this issue, please reopen if there's still open questions. Thank you for your patience on this issue. |
|
Great addition to google-cloud-storage, thank you @frankyn! |
|
Closed by #7091 |
|
Thanks @chvreddy for letting us know, it's really great to hear that you were unblocked! |
|
How can Workload Identity be used in conjunction with ActiveStorage - Google Cloud Storage? |
|
Hi @stefanahman, thanks for raising your question! Could you start a new issue with more background on your use case to better support you? |
|
@stefanahman Did you end up creating an issue and/or figuring it out? I implemented this signing code and that works, but now I have an implementation dilemma since Google Cloud Storage is only used in production, not tests or development. |
No, sorry. I haven't looked into it more. test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %> |
|
This issue still persists in 2023, workload identity is now the prevailing solution in GKE and likely other cloud based kubernetes solutions. What is the status on this? |
Environment details
Steps to reproduce
I've also logged in using
gcloud auth loginwith the same results.Code example
Replace 'some_bucket_name' with a bucket you control
The documentation for application default credentials suggests that this should work: https://cloud.google.com/iam/docs/service-accounts#application_default_credentials
I've also looked at https://googleapis.dev/ruby/google-cloud-storage/latest/file.AUTHENTICATION.html#cloud-sdk which again suggests that this should work for local development.
I know from the documentation that a
SignedUrlUnavailableerror is raised when 'when File#signed_url is unable to generate a URL due to missing credentials needed to create the URL.' However I can't find documentation that indicates how to create the required key.Can the error class please be updated to point to relevant documentation?
The text was updated successfully, but these errors were encountered: