Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error trying to use gcs storage plugin #18

Open
jairav opened this issue Feb 13, 2024 · 3 comments
Open

Error trying to use gcs storage plugin #18

jairav opened this issue Feb 13, 2024 · 3 comments

Comments

@jairav
Copy link
Contributor

jairav commented Feb 13, 2024

Attempting to use the new gcs storage plugin using the default provider options. Running on Snakemake 8.4.7, python 3.12.1.

This seems to be an issue with my argument. The bucket exists, authenticates off gcloud cli, and works fine on Snakemake 7.32.4

I have tried to remove the dash from the bucket name but it gives the same error.

(snakemake) % snakemake --executor kubernetes --software-deployment-method conda --default-storage-provider gcs --default-storage-prefix test-bucket --storage-gcs-project test-sandbox  
WorkflowError:
Error applying default storage provider gcs. Make sure to provide a valid --default-storage-prefix (see https://snakemake.github.io/snakemake-plugin-catalog/plugins/storage/{provider}.html). {validation_res}
@jairav
Copy link
Contributor Author

jairav commented Feb 13, 2024

Also worth noting that Error message URL does not resolve.

@vsoch
Copy link
Collaborator

vsoch commented Feb 13, 2024

Looks like it's missing a format string! I was never able to understand the logic of these plugins (and get this one to work myself) so we might need to tag team with @johanneskoester. I've been using aws s3 instead.

@jairav
Copy link
Contributor Author

jairav commented Mar 5, 2024

I think I have a solution. @vsoch It appears to just be that the query string used is "gcs://" where it should be "gs://"

I have created a fork of the plugin here which works on a simple workflow:


rule all:
    input:
        storage.gcs("gs://bkt-test-tmp-100/blah.txt")
    output:
        storage.gcs("gs://bkt-test-tmp-100/scratch/SUCCESS.txt")
    shell:
        """
        cp {input} {output} 
        """

I can create a PR with this fork. I also updated the pyproject.toml file as mentioned in #19 .

vsoch pushed a commit that referenced this issue Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants