feat: Add configurable protocol support to ensembl reference download #2649

pettyalex · 2024-02-15T22:12:22Z

This PR adds support for configurable protocol, ftp, http, or https to ensembl reference data download and vep cache download.

I was evaluating snakemake-wrapper and widely used workflows as a tool for my group to potentially use, and immediately ran into our firewall rules. By default we're able to make outgoing HTTP and HTTPS requests, but not FTP. I can request a change to our firewall, but it also would be helpful if these rules that download reference files were able to download them over HTTP, as ftp.ensembl.org has has been available over HTTP for a very long time.

If you'd prefer, I can create an issue to track this. I also intend to update test cases on these wrappers, and if you are happy with this pattern I could apply it across all wrappers in this repository that are currently hard-coded to make ftp requests.

QC

I confirm that:

For all wrappers added by this PR,

there is a test case which covers any introduced changes,
input: and output: file paths in the resulting rule can be changed arbitrarily,
either the wrapper can only use a single core, or the example rule contains a threads: x statement with x being a reasonable default,
rule names in the test case are in snake_case and somehow tell what the rule is about or match the tools purpose or name (e.g., map_reads for a step that maps reads),
all environment.yaml specifications follow the respective best practices,
the environment.yaml pinning has been updated by running snakedeploy pin-conda-envs environment.yaml on a linux machine,
wherever possible, command line arguments are inferred and set automatically (e.g. based on file extensions in input: or output:),
all fields of the example rules in the Snakefiles and their entries are explained via comments (input:/output:/params: etc.),
stderr and/or stdout are logged correctly (log:), depending on the wrapped tool,
temporary files are either written to a unique hidden folder in the working directory, or (better) stored where the Python function tempfile.gettempdir() points to (see here; this also means that using any Python tempfile default behavior works),
the meta.yaml contains a link to the documentation of the respective tool or command,
Snakefiles pass the linting (snakemake --lint),
Snakefiles are formatted with snakefmt,
Python wrapper scripts are formatted with black.
Conda environments use a minimal amount of channels, in recommended ordering. E.g. for bioconda, use (conda-forge, bioconda, nodefaults, as conda-forge should have highest priority and defaults channels are usually not needed because most packages are in conda-forge nowadays).

…embl reference data download and vep cache download.

fgvieira · 2024-02-16T07:06:34Z

Thanks for your contribution.
Can you add a test case for (e.g.) HTTP and HTTPS?

pettyalex · 2024-02-19T20:33:40Z

I've run snakefmt, black and the linter over these changes. I believe this to be ready to merge.

fgvieira · 2024-02-20T18:38:49Z

@pettyalex can you fix the fail test?
And you need to add the tests to test.py.

…n. Add each new test case to test.py.

pettyalex · 2024-02-22T16:12:20Z

@fgvieira I fixed all the tests, and now see that what I had previously done introduced multiple ways to generate the desired output files.

I also unintentionally ran black over test.py, which it seems like is not being done? Should I revert running black over test.py, or should we have it running through black as well?

fgvieira · 2024-02-23T22:11:22Z

test.py

+def test_vep_cache_https_protocol():
+    run(
+        "bio/vep/cache",
+        ["snakemake", "--cores", "1", "resources/vep/cache", "--use-conda", "-F"],


Suggested change

["snakemake", "--cores", "1", "resources/vep/cache", "--use-conda", "-F"],

["snakemake", "--cores", "1", "resources/vep/cache", "--use-conda", "-F", "-s", "vep_cache_https.smk"],

fgvieira · 2024-02-23T22:11:36Z

test.py

+            "-s",
+            "vep_cache_https.smk",


Suggested change

"-s",

"vep_cache_https.smk",

fgvieira · 2024-02-23T22:16:36Z

test.py

@@ -5428,17 +5461,26 @@ def test_ensembl_variation_with_contig_lengths():


 @skip_if_not_modified
-def test_ega_fetch():
+def test_ensembl_variation_old_release():


Suggested change

def test_ensembl_variation_old_release():

def test_ensembl_variation_old_release_https_protocol():

johanneskoester

Fantastic, thanks! Can you please also add the respective protocol lines to each Snakefile in the test folder? Those are used to generate the wrapper docs, from which people can copy paste into their workflows.

Allow for custom URLs (fix issues #366 and #2649). ### QC  * [x] I confirm that: For all wrappers added by this PR, * there is a test case which covers any introduced changes, * `input:` and `output:` file paths in the resulting rule can be changed arbitrarily, * either the wrapper can only use a single core, or the example rule contains a `threads: x` statement with `x` being a reasonable default, * rule names in the test case are in [snake_case](https://en.wikipedia.org/wiki/Snake_case) and somehow tell what the rule is about or match the tools purpose or name (e.g., `map_reads` for a step that maps reads), * all `environment.yaml` specifications follow [the respective best practices](https://stackoverflow.com/a/64594513/2352071), * the `environment.yaml` pinning has been updated by running `snakedeploy pin-conda-envs environment.yaml` on a linux machine, * wherever possible, command line arguments are inferred and set automatically (e.g. based on file extensions in `input:` or `output:`), * all fields of the example rules in the `Snakefile`s and their entries are explained via comments (`input:`/`output:`/`params:` etc.), * `stderr` and/or `stdout` are logged correctly (`log:`), depending on the wrapped tool, * temporary files are either written to a unique hidden folder in the working directory, or (better) stored where the Python function `tempfile.gettempdir()` points to (see [here](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir); this also means that using any Python `tempfile` default behavior works), * the `meta.yaml` contains a link to the documentation of the respective tool or command, * `Snakefile`s pass the linting (`snakemake --lint`), * `Snakefile`s are formatted with [snakefmt](https://github.com/snakemake/snakefmt), * Python wrapper scripts are formatted with [black](https://black.readthedocs.io). * Conda environments use a minimal amount of channels, in recommended ordering. E.g. for bioconda, use (conda-forge, bioconda, nodefaults, as conda-forge should have highest priority and defaults channels are usually not needed because most packages are in conda-forge nowadays).

fgvieira · 2024-05-28T12:09:25Z

PR #2928 allowed for custom URLs when downloading from VEP cache. Maybe something similar can be done here for ensembl annotation/variation/sequence?

Add the support for configurable protocol, ftp, http, or https to ens…

16e9da4

…embl reference data download and vep cache download.

pettyalex added 2 commits February 19, 2024 14:21

Add tests for getting reference data over http

d82fa78

Run snakefmt over reference download tools.

94cdbfd

Split off https into different snakemake files to avoid name collisio…

17d4d28

…n. Add each new test case to test.py.

Fix tests for annotation and make https protocol output gzipped.

1a84082

fgvieira reviewed Feb 23, 2024

View reviewed changes

test.py

Comment on lines +5865 to +5866

"-s",

"vep_cache_https.smk",

Copy link

Collaborator

fgvieira Feb 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

"-s",

"vep_cache_https.smk",

fgvieira reviewed Feb 23, 2024

View reviewed changes

johanneskoester requested changes Feb 27, 2024

View reviewed changes

fgvieira mentioned this pull request May 13, 2024

fix: issue #366 and #2649 #2928

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add configurable protocol support to ensembl reference download #2649

feat: Add configurable protocol support to ensembl reference download #2649

pettyalex commented Feb 15, 2024 •

edited

fgvieira commented Feb 16, 2024

pettyalex commented Feb 19, 2024

fgvieira commented Feb 20, 2024

pettyalex commented Feb 22, 2024

fgvieira Feb 23, 2024

fgvieira Feb 23, 2024

fgvieira Feb 23, 2024

johanneskoester left a comment

fgvieira commented May 28, 2024

	["snakemake", "--cores", "1", "resources/vep/cache", "--use-conda", "-F"],
	["snakemake", "--cores", "1", "resources/vep/cache", "--use-conda", "-F", "-s", "vep_cache_https.smk"],

	def test_ensembl_variation_old_release():
	def test_ensembl_variation_old_release_https_protocol():

feat: Add configurable protocol support to ensembl reference download #2649

Are you sure you want to change the base?

feat: Add configurable protocol support to ensembl reference download #2649

Conversation

pettyalex commented Feb 15, 2024 • edited

QC

fgvieira commented Feb 16, 2024

pettyalex commented Feb 19, 2024

fgvieira commented Feb 20, 2024

pettyalex commented Feb 22, 2024

fgvieira Feb 23, 2024

Choose a reason for hiding this comment

fgvieira Feb 23, 2024

Choose a reason for hiding this comment

fgvieira Feb 23, 2024

Choose a reason for hiding this comment

johanneskoester left a comment

Choose a reason for hiding this comment

fgvieira commented May 28, 2024

pettyalex commented Feb 15, 2024 •

edited