[GH-1641] WFL Release v0.18.0 #599
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Snapshot creation uses datetime rather than date * Making the name of the column used for datetime interval variable Co-authored-by: rhiananthony <ranthony@broadinstitute.org>
…ultiple times (#358) [GH-1278] Fix SG update-workload! implementation from trying to update the clio BAM records for each Succeeded workflow repeatedly by restoring the :finished guard for register-workload-in-clio. Introducted in https://broadinstitute.atlassian.net/browse/GH-1277.
…Release (#351) * Restore inputs processing hacks. * GDCWholeGenomeSomaticSingleSample moved. * Run off of /develop/ branch instead. * clojure -M:format * Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release. * Remove the Rich Comment.
* Draft doc module for SG. * Update navigation Camel^H^H^H^H^HYAML.
cherry-pick from main
The `Dockerfile` is being invoked in the root directory - we don't need to add the parent directory, just the current one.
Catch all throwables to make sure a workload failure doesn't bring down workflow-launcher.
* util/do-or-nil-silently should move to build.clj * Fixed indentation to match standard
* First pass: create-submissions generates one submission per entity. * Cleaned up TODOs for first pass attempt, reformatted tests to pass lint step in build. * Second pass: support entity set creation when specifying >1 entity for a submission. Refactored bigquery table dump to pull out now-common code. * Address PR feedback To simplify, will generate an entity set even for the singleton input. Removed firecloud/consolidate-entities-to-set as a result. Need to pass in 'expression' to submission creation payload when specifying an entity set. Supplemented integration and unit tests.
* Create an interface all hornets work on top of. * Apply suggestions from code review * Update readme.md Build Boad.
…ty (#368) * Boad -=> Board * Still doesn't round-trip on doubles and values like "19A". * Unit test wfl.tsv. * Ensure TSV mappability. * Ensure TSV is tabulatable. * Fix bug in assert-mapulatable!.
We create the snapshot in the TDR, and link to it in the workspace via Rawls.
…ema (#375) * Add snapshot_reference_id to Sarscov2IlluminaFull table schema Because schema update hasn't run, altering existing table definition in place. * Documentation changes from local Postgres debugging, onboarding - Clarified order of operations when installing, using local Postgres - Fixed broken intra-doc links - Added docs readme with instruction for launching local documentation site * Add instruction for recreating wfl DB to development docs * Remove references to undefined documentation files
Simply getting the submission again is not sufficient to guarantee that the workflow has been queued for execution. Poll instead.
* Update documentation and infrastructure for release candidates - creating release candiates - bashing release candidates Restore version override in cli.py. Re-tag latest docker images in cli.py so that version can be overwritten. Update IMAGES target in Makefile to add the :latest tag. Clean up :latest images on distclean target. * fix whitespace
Bumps [ssri](https://github.com/npm/ssri) from 6.0.1 to 6.0.2. - [Release notes](https://github.com/npm/ssri/releases) - [Changelog](https://github.com/npm/ssri/blob/v6.0.2/CHANGELOG.md) - [Commits](npm/ssri@v6.0.1...v6.0.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add interactive script ahead of covid-19 demo tomorrow * fix rawls test failure * reformat test * share demo remources with cdc-covid-surveillance firecloud group * corrections from review * format for consistency * clone a dev workspace instead of the prod one * fix test-import-snapshot - workspace changed * fix arrays test failures
* changelog changes * Link workload with source, sink and executor tables. * Update the p-keys to bigint. * SQL has reserved table. * Initial creat covid workload commit * Verifies dataset as well * Update DB schema. * Address comments. * v0.6.0 Release (#359) * [GH-1278] Fix SG update-workload! implementation from updating Clio multiple times (#358) [GH-1278] Fix SG update-workload! implementation from trying to update the clio BAM records for each Succeeded workflow repeatedly by restoring the :finished guard for register-workload-in-clio. Introducted in https://broadinstitute.atlassian.net/browse/GH-1277. * GH-1188: Bump GDCWholeGenomeSomaticSingleSample Version When Lantern Release (#351) * Restore inputs processing hacks. * GDCWholeGenomeSomaticSingleSample moved. * Run off of /develop/ branch instead. * clojure -M:format * Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release. * Remove the Rich Comment. * GitHub `develop` infrastructure + release changes (#356) Update test actions to only run on PRs. Update release action to only run on new commits into main. Update + format docs about branching off develop and releasing into main. Restore tag-and-push-images in cli.py * GH-1282: Document WFL's support for Somatic Genomes. (#361) * Draft doc module for SG. * Update navigation Camel^H^H^H^H^HYAML. * Update CHANGELOG.md for v0.6.0, including previous patches Co-authored-by: Tom Lyons <tbl3rd@gmail.com> Co-authored-by: Tom Lyons <tbl@broadinstitute.org> * Resolve merge conflicts. * Fix issues generated from rebasing. * Fix the failing integration tests. Co-authored-by: rhiananthony <ranthony@broadinstitute.org> Co-authored-by: Edmund Higham <ehigham@users.noreply.github.com> Co-authored-by: Tom Lyons <tbl3rd@gmail.com> Co-authored-by: Tom Lyons <tbl@broadinstitute.org>
* Add multimethods for source, executor and sink operations Add skeleton implementations for tdr-source * make use of `!` more consistent? * add create operations for source, executor and sink
- Data model change: a snapshot_reference_id is linked to a workspace and should be associated with the executor, not the source (TDR in our use case) - Added covid/get-imported-snapshot-reference - nil or snapshot reference from Rawls for snapshot_reference_id in executor details instance (to be created from TerraExecutorDetails) - Added covid/import-snapshot! - import snapshot to workspace, writing to DB if successful - Added integration tests (incomplete coverage)
RR: https://broadinstitute.atlassian.net/browse/GH-1295 firecloud's flexibleImportEntities has a size limit on the TSV file you POST. Unfortunately, one sarscov2_illumina_full workflow's outputs exceeds this. We can work around this issue by using Rawl's batchUpsert. This is slightly different as it takes a list of operations on how to construct the entity rather than the entity serialised to TSV. In this PR, I've demonstrated how we can use Rawls to import the workflow's outputs into the workspace. I've also updated the demo to do this for a workflow that has already passed.
…them (#372) * Simplify day intervals. * Document wfl.tools.snapshots. * Map name over keyword arguments. * Force the production Rawls. * Checkpoint groups getters. * Patch rebase conflict. * Implement start-covid-workload! kinda. * Checkpoint start-covid-workload! test. * Add covid-workload-request to support unit. * clojure -M:format * Move Rich comment into a unit test. * Respond to comments and tidy up a little * Remove unit test now covered by integration. * Document clojure.test/test-vars to remind me.
Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.4.7 to 1.5.1. - [Release notes](https://github.com/unshiftio/url-parse/releases) - [Commits](unshiftio/url-parse@1.4.7...1.5.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update db schemas. * Make the names more clear. * Address comments. * Switch to SQL syntax from XML. * Address more comments. * one more update.
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/yargs/y18n/releases) - [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md) - [Commits](https://github.com/yargs/y18n/commits) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ur (#386) RR: https://broadinstitute.atlassian.net/browse/GH-1301 Introduce wfl.util.UserVisibleException - a new type of exception that we should handle to throw and handle errors that users are meant to see. Re-wire the background loop to handle UserVisibleException and also be more exception safe. Start to plumb in email notifications.
* Set the API's image pullPolicy to Always. * Bump another random "semantic" version number.
* Update reitit. * Reduce cleverness. * Expose the logging_level API. * Add logging_level endpoints. * Fix bug in level-string? ... jeez. * Add logging_level endpoint test. * clojure -M:format` in this directory. * Respond to OK's comments.
And remove unnecessary module imports in `wfl.api.handlers`: previously WFL would complain about invalid pipelines when creating workloads without these imports, but this no longer seems to be the case.
Define concise, consistent workload representations for logging. Integrate into server, slack, and source (for staged workloads) logs.
…-filters. (#583) * Look at only the first 7 workloads returned. * Simplify.
Slack token now stored in environment variable, and propagated from Vault to GitHub Secrets via Atlantis.
…oud (#585) And log warning instead of error on method config version mismatch.
GH-1629 TerraExecutor gets method configuration version from Firecloud (#585) GH-1553 GH-1604 Add error handling to Slack service (#584) GH-1628 Look at the first 7 workloads returned in test-workflows-by-filters. (#583) GH-1617 Consistent workload representation for logs (#579) GH-1624 Rename covid -> staged in code, docs (#580)
Now Slack notifications are always enabled.
#591) And fix failing Firecloud integration tests by referencing a new unarchived workflow.
Instead, we can rely on temporary datasets which automatically delete any of their snapshots before the dataset.
tbl3rd
approved these changes
Mar 31, 2022
okotsopoulos
added a commit
that referenced
this pull request
Apr 4, 2022
GH-1623 workloads/update-workload! should define own transactions (#597) GH-1640 Broken Terra Submission links in completed workflow Slack messages (#598) GH-1639 Pin jinja2 version for mkdocks compatibility (#596) GH-1433 No TDR temporary snapshots in tests (#595) GH-1619 Add workload info to sink logs (#594) GH-1638 Disable link unfurling when Slacking watchers (#593)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Create WFL v0.18.0 release with small Slack fixes / improvements, and all of the rest of my code before I transition fully to Jade Core team.
Changes
System Tests
Passed:
Review Instructions
@sjacobs146 -- please read through this section in its entirety: https://broadinstitute.atlassian.net/wiki/spaces/~206513899/pages/3020554245/WFL+v0.10.x+release+and+deployment+notes#:check_mark:-Release-v0.10.0
I made a few updates to it so that it reflects what I did to generate this release branch and PR.
Most notable is that I explicitly point out the funny smells and manual interventions I needed to take along the way!
We never had time to prioritize untangling our "gitflow" release process / squash merge issues that result in all of the commits from WFL v0.7.0 onward showing up in the Github history of the release :( we usually say, please disregard the commit list when reviewing a release PR which isn't great.
A note on merging release PRs
The "squash merge" behavior by default lists all 200+ commits as its commit message:
I copy the contents of the v0.18.0 changelog as the commit message instead: