Update Qurro to support pandas v1 and up, and thus newer versions of …

…QIIME 2 (#322)

* DEP: Update setup.py re: python and pandas #315

* DEV: port CI from Travis to GH Actions: close #316

* TST: For now, omit "make notebooks" from CI

Maybe we can make another GitHub Actions for these later; but
Songbird is causing tensorflow nonsense to pop up, and this is not
the sort of thing I think we should spend time fixing (esp with
the advent of birdman)

* DEP: pin min biom vsn and add some comments

* DEP: Fix biom_table_to_sparse_df for pandas >= 1

See #258 and #315. not confident this is done yet (and if nothing
else the rest of the code gleefully refers to "SparseDataFrame"
because 2019 marcus was a schmuck), but this at least fixes a fair
amount of failing tests

* DEP: remove some warnings, docs, fix a test re: pd

* TST: Fix the python tests!!! #258

The problem was using .loc[] on these sparse dataframes. whoops

* STY: tiny style fixes

* DEP: knock out some pandas warnings

* DEP: np.matrix() -> np.array() in qarcoal tests

since apparently it's deprecated, or about to be deprecated, idk

* DEP/STY: Fix more warnings; remove unused import

most of these warnings were just pd.DataFrame.append() being
deprecated and replaced with pd.concat()

* DOC: one of the demos' JS data slightly changed

looks like it's a tiny floating-point thing -- probably an artifact
of working here on a new operating system, on a new python version,
a new pandas version, a new biom version, etc. shouldn't make a
noticeable difference

* DOC: update readme re: min Q2 vsn

* TST: matrix of qiime 2 versions

nice!

* TST: more detailed comment about Q2 vsn matrix

* DOC: remove the "Sparse" from "SparseDataFrame"

* REL: version kick

* TST: Add standalone CI

IIRC something about how our specific altair version works makes it
incompatible with python 3.10. let's test that here -- if needed,
we can update the README to disallow python versions >= 3.10. (And
then we can look into removing the altair pin when absolutely needed.)

* TST: attempt to get standalone tests working

* TST: attempt to fix pytest q2 exclusion

* DEP: ok py 3.10 is a no go

* STY: fix formatting

* DOC: Rerun 4 / 6 example notebooks

Songbird and ALDEx2 ones will cause problems

* DOC: tidy/update readme refs

* DOC: update jake fish dataset ref on website

* DOC: Fix songbird notebook!, standardize output rm

* BLD: rm (now-)unused comments from q2 ci

* DOC: fix transcriptomics ntbk :)

* REL: update changelog

* REL: update changelog

* TST: see if we can finagle q2 2020.6 / 2020.8?

since i thiiiink these versions mighta worked with the pandas >= 1
syntax

that being said, i don't think it makes sense to devote time/energy
to officially supporting them; just wanna check

* TST: remove Q2 2020.6 / 2020.8 in CI

Looks like the tests themselves pass for these versions, but the
style-checking with black fails due to incompatibility with click.

yeah this is enough for me to not bother supporting these versions
imo

* DOC: songbird compatibility deets

* DEV/DOC: update dev docs re: 2022

the apocalypse came and all i got was this pull request

* REL: update changelog about updating contributing

about about about about aboot

* REL: minor chglog tidying

.github/workflows/qiime2.yml

@@ -0,0 +1,64 @@
+# Based on
+# https://github.com/fedarko/strainFlye/blob/main/.github/workflows/main.yml
+# (which was based on EMPress' workflow); some of the Node.js stuff based on
+# MetagenomeScope's JS workflow
+name: QIIME 2 CI
+on: [push, pull_request]
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        # Test on multiple QIIME 2 versions. This is a nice way of making sure
+        # that Qurro works on all of the QIIME 2 versions it actually, you
+        # know, supports.
+        #
+        # We could use a matrix of just QIIME 2 version strings (e.g.
+        # "2020.11", ...), but other stuff about the URLs change across QIIME 2
+        # versions (e.g. the "-py36-" or "-py38-" thing). It ends up being
+        # easier just have the variable here be the URLs to the YAMLs. That
+        # sentence has too many acronyms. Look, you get the point.
+        #
+        # Worth noting: the final of these URLs (with qiime2-latest) will
+        # always be set to the latest version of QIIME 2. Thanks @thermokarst
+        # for the heads up about this :)
+        q2-yaml: [
+          "https://raw.githubusercontent.com/qiime2/environment-files/master/2020.11/release/qiime2-2020.11-py36-linux-conda.yml",
+          "https://raw.githubusercontent.com/qiime2/environment-files/master/latest/staging/qiime2-latest-py38-linux-conda.yml"
+        ]
+
+    steps:
+
+      # Updating from v2 to v3 to match, as of writing,
+      # https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+      - name: Check out code
+        uses: actions/checkout@v3
+
+      - name: Download the QIIME 2 conda environment YAML
+        run: wget -O q2-env.yml ${{ matrix.q2-yaml }}
+
+      # https://github.com/conda-incubator/setup-miniconda#example-3-other-options
+      - name: Hit the yoinky sploinky (install QIIME 2)
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          activate-environment: qiime2-dev
+          environment-file: q2-env.yml
+
+      - name: Install Qurro (and its pip dependencies)
+        run: conda run -n qiime2-dev pip install -e .[dev]
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v1
+
+      - name: Install Qurro's Node.js development dependencies
+        run: npm install -g mocha-headless-chrome nyc prettier@2.0.5 jshint
+
+      - name: Run tests
+        run: conda run -n qiime2-dev make test
+
+      - name: Lint and stylecheck
+        run: conda run -n qiime2-dev make stylecheck
+
+      - name: Upload code coverage information to Codecov
+        uses: codecov/codecov-action@v2

.github/workflows/standalone.yml

@@ -0,0 +1,44 @@
+# Based on
+# https://github.com/fedarko/pyfastg/blob/master/.github/workflows/main.yml
+name: Standalone CI
+on: [push, pull_request]
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8, 3.9]
+
+    steps:
+
+      - name: Check out code
+        uses: actions/checkout@v3
+
+      # https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install Cython and NumPy
+        run: pip install cython "numpy >= 1.12.0"
+
+      - name: Install Qurro (and its pip dependencies)
+        run: pip install -e .[dev]
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v1
+
+      - name: Install Qurro's Node.js development dependencies
+        run: npm install -g mocha-headless-chrome nyc prettier@2.0.5 jshint
+
+      # Just run the tests; don't bother running stylechecking / linting, since
+      # that's already done in the QIIME 2 CI.
+      - name: Run tests
+        run: |
+          make pytest_standalone
+          make jstest
+
+      - name: Upload code coverage information to Codecov
+        uses: codecov/codecov-action@v2

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Qurro changelog
 
+## Qurro 0.8.0 (date TBD)
+### Features added
+- Add an option to Qurro's visualization interface to exclude metadata columns
+  from the "sample plot data" output TSV
+  ([#306](https://github.com/biocore/qurro/issues/306),
+  [#313](https://github.com/biocore/qurro/pull/313)).
+  - This should make it easier to merge the sample log-ratios exported from
+    Qurro with a sample metadata file -- since now these metadata columns won't
+    be duplicated, which would cause problems. (This problem comes up in
+    [Gemelli](https://github.com/biocore/gemelli)'s tutorials.)
+### Backward-incompatible changes
+- Switched the required pandas version from `>= 0.24.0, <1` to `>= 1`.
+  This makes Qurro now compatible with QIIME 2 versions `>= 2020.11`!
+  - However, this change removes support for older QIIME 2 versions. (It may be
+    possible to get this version of Qurro installed into an older QIIME 2
+    environment, but we do not explicitly support this.)
+  - Notably, Songbird has not yet made the shift to pandas `>= 1`, so Songbird
+    and Qurro will need to be installed into separate environments; the "Red
+    Sea" example notebook details this process.
+### Bug fixes
+### Performance enhancements
+### Miscellaneous
+- Improve various parts of Qurro's code to remove various warnings (for
+  example, about certain things being deprecated).
+- Update Qurro's example notebooks:
+  - Updated the "Red Sea" notebook to explain how to use
+    Songbird and Qurro in different conda environments.
+  - Updated the "ALDEx2" notebook to explain how we recommend installing ALDEx2
+    nowadays.
+- Updated the documentation to refer to the published version of the Mackerel
+  data's paper ([Minich et al. 2020](https://journals.asm.org/doi/full/10.1128/mSphere.00401-20)).
+- Ported Qurro's continuous integration from Travis CI to GitHub Actions
+  ([#316](https://github.com/biocore/qurro/issues/316)).
+- Improved Qurro's continuous integration in multiple ways:
+  - Test on multiple QIIME 2 versions.
+  - Test the standalone Qurro functionality on multiple Python versions.
+  - Test the standalone Qurro functionality in a non-QIIME-2 environment
+    (similar to EMPress' "standalone" CI).
+- Updated the development documentation regarding the minimum QIIME 2 version,
+  incompatibilities between `black` and `click`, etc.
+
 ## Qurro 0.7.1 (May 22, 2020)
 ### Features added
 ### Backward-incompatible changes

CONTRIBUTING.md

@@ -14,12 +14,12 @@ in this repository or email the Qurro development team
 1. Clone your fork of Qurro's source code to your computer.
 2. Create a development conda environment for Qurro:
     1. Install the latest version of QIIME 2 natively,
-       [as you would normally](https://docs.qiime2.org/2019.7/install/native/).
-       _You'll need to install a QIIME 2 version of at least 2019.7._
+       [as you would normally](https://docs.qiime2.org/2022.2/install/native/).
+       _You'll need to install a QIIME 2 version of at least 2020.11._
     2. In a terminal, navigate to the folder to which you cloned your fork of
        Qurro's source code above. Run `pip install -e .[dev]` inside this folder to
        install Qurro along with its normal and development Python dependencies.
-    4. Install the various Node.js requirements for testing Qurro's JavaScript
+    3. Install the various Node.js requirements for testing Qurro's JavaScript
        code. This can be done by running
        `npm install -g mocha-headless-chrome jshint prettier@2.0.5 nyc`. Note that
        this will install these programs globally on your system.
@@ -53,6 +53,23 @@ generated by Qurro's Python tests, and is used to populate some of the
 JavaScript tests. Long story short, you'll just need to run `make test` (or
 just `make pytest`) before running `make jstest`.
 
+### When running QIIME 2, Qurro, etc. I get `TypeError: __init__() got multiple values for argument 'obj'`
+
+This happens due to a [Click](https://click.palletsprojects.com/en/8.1.x/)
+version being installed that is unsupported by
+QIIME 2. See [this q2cli issue](https://github.com/qiime2/q2cli/issues/259) for
+context. This can come up in development because one of Qurro's development
+dependencies is
+[Black](https://github.com/psf/black), and later versions of Black can rely
+on Click 8 (and as of writing QIIME 2 doesn't seem to support Click 8 yet).
+
+If you run into this error, you can probably get around it by installing an
+older version of Black, although earlier versions of Click might disagree with
+the later versions that we use in the CI. There isn't a great solution for this
+yet that I know of (short of pinning Black to an earlier version and
+reformatting all the Python code accordingly...), but hopefully when QIIME 2
+supports Click 8 the problem will vanish.
+
 ## Acknowledgements
 
 This document was loosely based on Emperor's [CONTRIBUTING.md file](https://github.com/biocore/emperor/blob/new-api/CONTRIBUTING.md).

Makefile

@@ -18,6 +18,14 @@ test: pytest jstest
 pytest:
 	python3 -B -m pytest qurro/tests --cov qurro
 
+# Ignore test files (and functions) that have q2 in the name
+# This way, we can run non-QIIME2-specific tests outside of a QIIME 2 conda
+# environment. (For some reason, -k "not q2" wasn't working, even though the
+# docs said that it should consider file names... well, this seems to work.)
+# https://stackoverflow.com/questions/11117062/how-to-tell-py-test-to-skip-certain-directories#comment113260122_41940247
+pytest_standalone:
+	python3 -B -m pytest qurro/tests --cov qurro --ignore-glob "*q2*"
+
 jstest:
 	@# Re-update specs for JS tests by running update_js_test_jsons.py
 	python3 qurro/tests/update_js_test_jsons.py

README.md

@@ -5,7 +5,8 @@
 </div>
 
 <div align="center">
-<a href="https://travis-ci.com/biocore/qurro"><img src="https://travis-ci.com/biocore/qurro.svg?branch=master" alt="Build Status" /></a>
+<a href="https://github.com/biocore/qurro/actions/workflows/qiime2.yml"><img src="https://github.com/biocore/qurro/actions/workflows/qiime2.yml/badge.svg" alt="QIIME 2 CI" /></a>
+<a href="https://github.com/biocore/qurro/actions/workflows/standalone.yml"><img src="https://github.com/biocore/qurro/actions/workflows/standalone.yml/badge.svg" alt="Standalone CI" /></a>
 <a href="https://codecov.io/gh/biocore/qurro"><img src="https://codecov.io/gh/biocore/qurro/branch/master/graph/badge.svg" alt="Code Coverage" /></a>
 <a href="https://zenodo.org/badge/latestdoi/158270494"><img src="https://zenodo.org/badge/158270494.svg" alt="DOI" /></a>
 <a href="https://pypi.org/project/qurro"><img src="https://img.shields.io/pypi/v/qurro.svg" alt="PyPI" /></a>
@@ -57,8 +58,8 @@ Qurro can be used standalone (as a Python 3 script that generates a
 folder containing a HTML/JS/CSS visualization) or as a
 [QIIME 2](https://qiime2.org/) plugin (that generates a QZV file that can be
 visualized at [view.qiime2.org](https://view.qiime2.org/) or by using
-`qiime tools view`). *Starting with Qurro v0.3.0, Qurro requires a QIIME 2
-version of at least 2019.7.*
+`qiime tools view`). *Starting with Qurro v0.8.0, Qurro requires a QIIME 2
+version of at least 2020.11.*
 
 Qurro should work with most modern web browsers. Firefox or Chrome are
 recommended.
@@ -84,7 +85,7 @@ can be viewed online [here](https://biocore.github.io/qurro/demos/red_sea/index.
 
 ## Installation and Usage
 
-You can install Qurro using [pip](https://pip.pypa.io/en/stable/) or [conda](https://docs.conda.io/en/latest/). In either case, a python version of at least 3.6 and less than 3.8 is required to use Qurro.
+You can install Qurro using [pip](https://pip.pypa.io/en/stable/) or [conda](https://docs.conda.io/en/latest/). In either case, a python version of at least 3.6 and less than 3.10 is required to use Qurro.
 
 ### Installing with `pip`
 ```bash
@@ -203,8 +204,7 @@ For python testing/style checking, Qurro uses
 [flake8](http://flake8.pycqa.org/en/latest/), and
 [black](https://github.com/ambv/black). You'll also need to have QIIME 2
 installed to run most of the python tests (your QIIME 2 version should be at
-least 2019.7, due to the `FeatureData[Differential]` type being merged into
-q2-types starting with this release).
+least 2020.11; this is the version which switched to pandas >= 1).
 
 For JavaScript testing/style checking, Qurro uses
 [Mocha](https://mochajs.org/), [Chai](https://www.chaijs.com/),
@@ -222,7 +222,7 @@ also.
 ### Data Sources
 
 The test data located in `qurro/tests/input/mackerel/` were exported from
-QIIME 2 artifacts in [this repository](https://github.com/knightlab-analyses/qurro-mackerel-analysis). These data are from Minich et al. 2019 [1].
+QIIME 2 artifacts in [this repository](https://github.com/knightlab-analyses/qurro-mackerel-analysis). These data are from Minich et al. 2020 [1].
 
 The test data located in `qurro/tests/input/byrd/` are from
 [this repository](https://github.com/knightlab-analyses/reference-frames).
@@ -291,22 +291,22 @@ Here's the BibTeX:
 ## References
 
 [1] Minich, J. J., Petrus, S., Michael, J. D., Michael, T. P., Knight, R., &
-Allen, E. E. (2019). Temporal, environmental, and biological
-drivers of the mucosal microbiome in a wild marine fish, Scomber japonicus.
-_bioRxiv_, page 721555. [Link](https://www.biorxiv.org/content/10.1101/721555v1).
+Allen, E. E. (2020). Temporal, environmental, and biological drivers of the
+mucosal microbiome in a wild marine fish, Scomber japonicus. _mSphere, 5_(3),
+e00401-20. [Link](https://journals.asm.org/doi/full/10.1128/mSphere.00401-20).
 
-[2] Byrd, A. L., Deming, C., Cassidy, S. K., Harrison, O. J., Ng, W. I., Conlan, S., ... & NISC Comparative Sequencing Program. (2017). Staphylococcus aureus and Staphylococcus epidermidis strain diversity underlying pediatric atopic dermatitis. _Science translational medicine, 9_(397), eaal4651.
+[2] Byrd, A. L., Deming, C., Cassidy, S. K., Harrison, O. J., Ng, W. I., Conlan, S., ... & NISC Comparative Sequencing Program. (2017). Staphylococcus aureus and Staphylococcus epidermidis strain diversity underlying pediatric atopic dermatitis. _Science Translational Medicine, 9_(397), eaal4651.
 [Link](https://www.ncbi.nlm.nih.gov/pubmed/28679656).
 
 [3] Thompson, L. R., Williams, G. J., Haroon, M. F., Shibl, A., Larsen, P.,
 Shorenstein, J., ... & Stingl, U. (2017). Metagenomic covariation along densely
-sampled environmental gradients in the Red Sea. _The ISME journal, 11_(1), 138.
+sampled environmental gradients in the Red Sea. _The ISME Journal, 11_(1), 138.
 [Link](https://www.ncbi.nlm.nih.gov/pubmed/27420030).
 
 [4] Tripathi, A., Melnik, A. V., Xue, J., Poulsen, O., Meehan, M. J., Humphrey, G., ... & Haddad, G. (2018). Intermittent hypoxia and hypercapnia, a hallmark of obstructive sleep apnea, alters the gut microbiome and metabolome. _mSystems, 3_(3), e00020-18.
 [Link](https://www.ncbi.nlm.nih.gov/pubmed/29896566).
 
-[5] Caporaso, J. G., Lauber, C. L., Costello, E. K., Berg-Lyons, D., Gonzalez, A., Stombaugh, J., ... & Gordon, J. I. (2011). Moving pictures of the human microbiome. _Genome biology, 12_(5), R50.
+[5] Caporaso, J. G., Lauber, C. L., Costello, E. K., Berg-Lyons, D., Gonzalez, A., Stombaugh, J., ... & Gordon, J. I. (2011). Moving pictures of the human microbiome. _Genome Biology, 12_(5), R50.
 [Link](https://www.ncbi.nlm.nih.gov/pubmed/21624126).
 
 ## License

docs/index.html

@@ -104,8 +104,9 @@ <h2>Demos</h2>
                 <td>Songbird</td>
                 <td><a href="demos/mackerel/index.html">Here</a></td>
                 <td>
-                    <a href="https://www.biorxiv.org/content/10.1101/721555v1"
-                        >Minich et al. 2019 (under review)</a
+                    <a
+                        href="https://journals.asm.org/doi/full/10.1128/mSphere.00401-20"
+                        >Minich et al. 2020</a
                     >; differentials generated
                     <a
                         href="https://github.com/knightlab-analyses/qurro-mackerel-analysis"