Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: declare associative arrays in bash scripts #1844

Merged
merged 1 commit into from Sep 2, 2022

Conversation

mbhall88
Copy link
Member

@mbhall88 mbhall88 commented Sep 2, 2022

Description

This PR adds a hotfix for bash scripts. I don't know if this is a Bash version issue or not, but when I tried to run a bash script on v4.4.20 (I tested the implementation of bash scripts with v5.1.16) the named variabes were playing up.

Here is an example to illustrate the problem (I've removed snakemake_config for brevity)

#!/usr/bin/env bash

snakemake_input=( [0]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/filtered/illumina/PRJEB25972/SAMEA1101594/ERR2510324/ERR2510324.filtered.fq.gz" [reads]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/filtered/illumina/PRJEB25972/SAMEA1101594/ERR2510324/ERR2510324.filtered.fq.gz" [1]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/download/illumina/run_info.tsv" [run_info]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/download/illumina/run_info.tsv" [2]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/tbprofiler/.db.built" [db]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/tbprofiler/.db.built" )
snakemake_output=( [0]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324/results/ERR2510324.results.json" [report]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324/results/ERR2510324.results.json" )
snakemake_params=( [0]="--txt --no_trim -p ERR2510324" [opts]="--txt --no_trim -p ERR2510324" [1]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324" [outdir]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324" )
snakemake_wildcards=( [0]="illumina" [tech]="illumina" [1]="PRJEB25972" [proj]="PRJEB25972" [2]="SAMEA1101594" [sample]="SAMEA1101594" [3]="ERR2510324" [run]="ERR2510324" )
snakemake_resources=( [0]="1" [_cores]="1" [1]="1" [_nodes]="1" [2]="1000" [mem_mb]="1000" [3]="1000" [disk_mb]="1000" [4]="/hps/scratch/lsf_tmpdir/hl-codon-113-02" [tmpdir]="/hps/scratch/lsf_tmpdir/hl-codon-113-02" )
snakemake_log=( [0]="/hps/nobackup/iqbal/mbhall/drprg/paper/logs/rules/tbprofiler_predict/illumina/PRJEB25972/SAMEA1101594/ERR2510324.log" )

set -euxo pipefail

# shellcheck disable=SC2154
exec 2> "${snakemake_log[0]}" # send all stderr from this script to the log file

# shellcheck disable=SC2154
reads="${snakemake_input[reads]}"

when this script was executed, it was failing with /hps/nobackup/iqbal/mbhall/drprg/paper/.snakemake/scripts/tmpbf4avsq5.tbprofiler_predict.sh: line 16: reads: unbound variable.

But when I explicitly declare the snakemake variables as associative arrays (declare -A), the script ran fine.

#!/usr/bin/env bash

declare -A snakemake_input=( [0]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/filtered/illumina/PRJEB25972/SAMEA1101594/ERR2510324/ERR2510324.filtered.fq.gz" [reads]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/filtered/illumina/PRJEB25972/SAMEA1101594/ERR2510324/ERR2510324.filtered.fq.gz" [1]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/download/illumina/run_info.tsv" [run_info]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/download/illumina/run_info.tsv" [2]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/tbprofiler/.db.built" [db]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/tbprofiler/.db.built" )
declare -A snakemake_output=( [0]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324/results/ERR2510324.results.json" [report]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324/results/ERR2510324.results.json" )
declare -A snakemake_params=( [0]="--txt --no_trim -p ERR2510324" [opts]="--txt --no_trim -p ERR2510324" [1]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324" [outdir]="/hps/nobackup/iqbal/mbhall/drprg/paper/results/amr_predictions/tbprofiler/illumina/PRJEB25972/SAMEA1101594/ERR2510324" )
declare -A snakemake_wildcards=( [0]="illumina" [tech]="illumina" [1]="PRJEB25972" [proj]="PRJEB25972" [2]="SAMEA1101594" [sample]="SAMEA1101594" [3]="ERR2510324" [run]="ERR2510324" )
declare -A snakemake_resources=( [0]="1" [_cores]="1" [1]="1" [_nodes]="1" [2]="1000" [mem_mb]="1000" [3]="1000" [disk_mb]="1000" [4]="/hps/scratch/lsf_tmpdir/hl-codon-113-02" [tmpdir]="/hps/scratch/lsf_tmpdir/hl-codon-113-02" )
declare -A snakemake_log=( [0]="/hps/nobackup/iqbal/mbhall/drprg/paper/logs/rules/tbprofiler_predict/illumina/PRJEB25972/SAMEA1101594/ERR2510324.log" )

set -euxo pipefail

# shellcheck disable=SC2154
exec 2> "${snakemake_log[0]}" # send all stderr from this script to the log file

# shellcheck disable=SC2154
reads="${snakemake_input[reads]}"

I'm not quite sure what the root cause of the problem is, but declaring the snakemake variables fixes the problem so that seems sufficient I guess?

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

@sonarcloud
Copy link

sonarcloud bot commented Sep 2, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

Copy link
Contributor

@johanneskoester johanneskoester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, seems acceptable. Thanks!

@johanneskoester johanneskoester changed the title fix: declare associative arrays fix: declare associative arrays in bash scripts Sep 2, 2022
@johanneskoester johanneskoester merged commit 90ae449 into snakemake:main Sep 2, 2022
johanneskoester pushed a commit that referenced this pull request Sep 26, 2022
🤖 I have created a release *beep* *boop*
---


##
[7.14.1](v7.14.0...v7.14.1)
(2022-09-23)


### Bug Fixes

* allocation of local ssds in k8s tests
([#1870](#1870))
([d0de4dc](d0de4dc))
* allow script directive to take pathlib Path
([#1869](#1869))
([12cdc96](12cdc96))
* catch errors in remote.AUTO provider list
([#1834](#1834))
([c613ed2](c613ed2))
* consistently use text output in conda shell commands and various
little fixes for failing test cases due to conda package changes
([#1864](#1864))
([4234fe7](4234fe7))
* declare associative arrays
([#1844](#1844))
([90ae449](90ae449))
* fix falsely triggered reruns if input files are obtained via
workflow.source_path()
([#1862](#1862))
([2dc2e6a](2dc2e6a))
* fixed typos
([#1847](#1847))
([a1e49b6](a1e49b6))
* k8s container volume mounts as list
([#1868](#1868))
([5c54df3](5c54df3))
* None type error when invoking Workflow object manually
([#1731](#1731))
([dc45ccb](dc45ccb))
* request disk_mb resource from k8s
([#1858](#1858))
([f68f166](f68f166))
* respect shebang lines in post-deploy scripts (see deployment docs)
([#1841](#1841))
([c26c4b6](c26c4b6))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants