V2 (#123)

* delete v1 * remove hidden files * Zamba v1 docs port (#108) * workflow tests * remove tests workflow from v2 * Great migration (#113) * copy everything * first round of deletion * more deletions * more deletions * put everything under zamba * remove old readme * replace zamba_algorithms with zamba * integrate makefile * changed files * updates from save dir overwrite * add gitignore * use gitignore from zamba algo * remove load metadata * remove old dirs * lint * remove load metadata * remove load_metadata test * fix datamodule tests * simplify time dist model since there is only one * flake8 * remove unused file * put back lost code * back to long name because that is was model is registered as * species is no longer on model * use models without species prefix and zamba backbone finetune params; * rename time dist head since there is only one * further remove species * use lstrip instead * passing tests * add missing underscore * update links to setup.cfg * specify branches in workflow * update setup.cfg * Update setup.cfg Co-authored-by: Peter Bull <pjbull@gmail.com> * Update setup.cfg Co-authored-by: Peter Bull <pjbull@gmail.com> Co-authored-by: Peter Bull <pjbull@gmail.com> * bug fix (#114) * Skip on failure, actually drop duplicates, clarify logging (#115) * skip on failure, actually drop duplicates, clarify logging * correct comment * only apply to root validators * First draft of zamba v2 docs (#112) * Update zamba predict in cli doc * Update zamba train in cli doc * Formatting * Start updating index * Update ungulate pic * update to chimp image * specify image sizes * update image sizes * add labels to images * Format * add labels to frames * note about top div * start updating inputs_outputs * tweaks to index * Use new vids in index * use full size images * remove some todo notes * comments to cli md * comments to cli md * comments to cli md * Add comments to index * Comments to inputs_outputs * Incorporate preliminary comments * Start working on install * Start working on algorithms * Start reorganizing * Redo quickstart to have a python section * Add basics of slowfast * Use better compressed images * Try and view with new TOC * Add more model info * Start configuration documentation * Add all configurations * Update TOC with configurations doc page * Add where output is saved * Add placeholders for user tutorials * v2 updates * Add info about yolox * Send us labeled videos * Make available models top level * TODO about model details * Better explanation of YAML v CLI args * Copy edits and formatting * Update install page * Add basic use to available models * Update mkdocs.yml * Start working on python package page * Make TOC work correctly * Add explanation of more of the parameter options * Start user tutorials * Update based on new default help text * Copy editing * Add where to reference for python package * Remove repetitive parameter explanation from cli.md * Update TOC names * Update tutorial names * Add python package and training to quickstart * copy edits * Status before big reorganization * Restructure * Write predict tutorial * Other updates for restructuring * YAML configs page * Update homepage index * Save path for CSV * Update tutorials * Update default video loading configs * Record of extra code, then will delete * Remove extra md files * Save path updates * delete more extra files * copy edits * Update args * TODO updates * New default save path * Update megadetector explanation * Better ffmpeg installation instructions * Fix checkpoint saving behavior * Typos * Add troubleshooting sections and format python code blocks better * Add video loading requirements to yaml-config.md * Improve default model descriptions * Begin incorporating feedback * consistent FFmpeg capitalization * index testing * index testing * More feedback * Add debugging page * yaml feedback * Seprate common advanced options page * beautiful magic tabs and more copy editing * Tabs on model page * Support training models with just two classes (#117) * two class metric updates * Update tests/conftest.py Co-authored-by: Emily Miller <ejm714@gmail.com> Co-authored-by: Emily Miller <ejm714@gmail.com> * 32 not 31 species (#118) * Add erdantic diagram (#121) * add erdantic diagram * add missing docs reqs * unpin * Write out splits + fix for videos that cannot be loaded (#119) * write out splits * write out zeros for videos that cannot be loaded * create missing save dir * typo * Add OSes and codecov to github actions (#124) * add oses and codecov * try w/o windows * msvc config * test with env var * Give up on Windows for now * [V2] Clean up dev dependencies (#125) * Remove unneeded dev dependencies * Split out docs and lint deps for faster installation Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> * [V2] Release and docs CI (#126) * Add make command to build distributions * Add --version command * Add README * Test built distributions; add failed build notify * Reorder config to keep build metadata together at the top * Rename docs-publish to docs-master * Rename docs-master to docs-latest * Add maintainers docs * Add release workflow * Add mike for versioned docs * Run docs-latest on push to v2 * Remove code tags in nav * Add __main__ for python -m entrypoint Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> * Get default video loader config for models (#127) * get default video loader for model * get default vlc if not specified * set save dir to tmp path so splits.csv is written to temp dir * keep vlc in so we can use for training * add evenly sample * use loguru * set num workers to 3 * remove cpu count * revert change * show ffmpeg error in debug mode * add link to wiki * Rename README.md to MAINTAINING.md (#128) * API Reference with mkdocstrings (#129) * Remove upload_models.py script * Add API Reference with mkdocstrings * Minor documentation tweaks to fix rendering * Use sections instead of auto-expand * Need to install zamba to use mkdocstrings * Put back upload_models.py * Wrap upload_models in if __name__ = '__main__' Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> * Resized test assets (#130) * add resized test assets * resampled to fps=1 * update labels per new videos * update with new video * fix failed test * start python tutorial * Update v2 docs with new changes (#132) * clarify labels filepath columns * start updating video loader config list * update configs * Update video organization reqs * video_width/height and num_workers * filepaths * default num_workers * update num_workers in CLI * writing out train_config and predict_config yamls * splits.csv * updates based on netlify preview * add screencast video demo * fix linting fail * updates for caching PR 131 * typo * use best terminal video from asciinema * specific explanation of frame_selection_height * correct default batch size * add frame_selection_width v model_input_width * talk about num_workers more * flake8 fix * PR feedback * update home page language * fix typo and change from sections to expand * reduce toc depth to 2 to allow expand * use megadetectorlite consistently * move api reference to end * add train data size recs * Update docs/docs/train-tutorial.md rephrase train data size rec Co-authored-by: Emily Miller <ejm714@gmail.com> * Enable nav index page for models * make contribute section header * Edit MDLite language * put yolox back in Co-authored-by: Emily Miller <ejm714@gmail.com> Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> * Expose final three models (#133) * add official models dir * remove tf events * move to outside template folder * simplify model mapping; set batch size default to 2 * move official models into zamba * add vlc for slowfast * update manifest for official models * update filename * update filepath * must import models for them to be in available_models * remove config details used for training so these can be the baseline * update config * add backbone finetune config that was used in case defaults change * update labels for time dist; remove slowfast until it is retrained * update per new model * formatting * helper command * already path * Clarify caching (#131) * set ability to turn off caching for prediction * reomve cache dir from cli * test cache dir is set but not used * rename to MODEL_CACHE_DIR for consistency * rename cache_dir to model_cache_dir * move video cache dir into configs * remove unneeded code as caching is off by default * rename to video_cache_dir for clarity * remove * do not support setting in configs * put back in settings * lint and such * rebase fix * put cache_dir and cleanup option on video laoder config * add tests for caching * get empty vlc if none is passed * put within func to avoid writing to real path * add logging * bug fix * reomve old change * loguru uses warning not warn * load dotenv in init * setup logger in init and rename to log_level for simplicity * cleanup does not change hash * fix test * lint * add regression test * do not use load_dotenv * Fix transform look up bug (#135) * separate out weight lookup from class based dict * fix lookup * set split propotions to none if using split in labels * tense * shorten * fix failing test * add updated slowfast model * add test for specifying checkpoint * address deprectation warnings * update templates to reflect final models * fix imports * lint * fix make docs-serve * Update link to models and contribute pages * update erdantic schema * Update references to other pages * Add links to final github source code to docs (#141) * add working links with v2 branch * use master branch instead of v2 * refer to api docs instead of github * Automated workflow for publishing models (#140) * WIP * update weight lookup and hash config to get model name * use auto workflow * auto workflow * add logging * rename to publish models * auto workflow * add code for handling finetuned models * add docs for adding a new model * checklist * update model name * put models in official models dir on s3 * add lookup function * auto workflow * use model dir which is clearer than ckpt * update docs * new slowfast model * rglob needs list * update euro config * only hash once and have function do a lookup * make this not model specific * clarify comments * remove checkpoint from train_config * docs * function to get only model params * use function to subset train config * update yamls based on exclude rather than include * only set checkpoint if not from_scratch, otherwise look up hparams * missing import * lint * checkpoint will now be none for training fromm scratch * remove space * docs-latest needs zamba (#143) * Add densepose model (#142) * Add densepose model * Update densepose install and testing * format * remove errant test yml * more generic codec * append coverage * Update README for v2 (#134) * screencast video demo * video not showing * video show method * autoplay vid * vid * updated terminal video * test video embedding * terminal video * Embed video in README * start updating text * full draft of text * delete vid and update text * editing based on page rendering * rephrase * make same as index.md * update monkey video * edit README * improve formatting * add make docs-setup * delete index.md raw page * hybrid README * update * updates * correct contribute link * add changelog from HISTORY.md * create changelog folder placeholder * delete old changelog md * Simplify caching (#145) * simplify caching * nest function * simplify * move * alphabetize * lint * rename to cache path * fix dataset * Change default backbone finetuning, auto_lr_find, and fix persistent workers bug (#148) * Default patience from 3 to 5 * Default for auto_lr_find to False * Change default unfreeze_backbone_at_epoch to 5 * Fix persistent_workers bug * Revert changes to templates * 496 docs review (#150) * add denspose and update save dir * commit two autogenerated files * changelog edits * contribute edits * WIP edits for models * finish edits for models and separate out densepose * full list of parks * config options * vscode format * move config guide into tutorials * extra options * windows not tested * tutorial edits * remove extra nb * put save path back * quickstart edits * vscode formatting * finish quickstart edits * train tutorial * add template section * logging * ffpmeg install in readme * finetuning * capitalization * remove ffmpmeg * date * add densepose * tweak * simplify * fix densepose video link * caps and tensorboard * add help * remove pythong piece since this is focused on yaml files * more tweaks * edit history not change log * copy edits * fix changelog * table * table * typo * alphabetize * table bug * table edits * Simplify save_dir and some directory -> dir renames (#151) * wip renames * renames in docs * readme * data dir renamme in docs * rename in code from data_directory to data_dir * maintaining update * fix capitalization * further updates * tweak * do not overwrite * add overwrite save dir * add overwrite save dir to config * update configs with all info * use full train configuration * only upload if does not exist * tests for save * overwrite param * better set up and test for overwrite * docs * update docs with overwrite * from overwrite_save_dir to overwrite * missed rename * remove machine specific from vlc * unindent so test actually runs * check for local and cached checkpoints * should be and * write out predict config before preds start like we do for train config * update all configs and use only first 10 digits of hash * dry run check after save is configured; more robust test * reorder * show save directory * copy edits * update template * fix test * lower case for consistency * fix test * Fix docs links (#155) * update readme links * update makefile Co-authored-by: Emily Miller <ejm714@gmail.com> Co-authored-by: ejm714 <emily@drivendata.org> Co-authored-by: Katie Wetstone <46792169+klwetstone@users.noreply.github.com> Co-authored-by: Jay Qi <2721979+jayqi@users.noreply.github.com> Co-authored-by: Jay Qi <jayqi@users.noreply.github.com> Co-authored-by: Katie Wetstone <klwetstone@gmail.com> Co-authored-by: Robert Gibboni <robert@drivendata.org>
drivendataorg · Oct 26, 2021 · 6f30ce7 · 6f30ce7
1 parent 69b397a
commit 6f30ce7
Show file tree

Hide file tree

Showing 206 changed files with 11,541 additions and 6,546 deletions.
diff --git a/.dockerignore b/.dockerignore
diff --git a/.flake8 b/.flake8
diff --git a/.github/MAINTAINING.md b/.github/MAINTAINING.md
@@ -0,0 +1,124 @@
+# Maintainers Documentation
+
+This page contains documentation for zamba maintainers.
+
+## Release Instructions
+
+To release a new version of `zamba`:
+
+1. Bump the version in [`setup.cfg`](https://github.com/drivendataorg/zamba/blob/master/setup.cfg). The version number should follow [semantic versioning](https://semver.org/).
+2. Create a new release using the [GitHub releases UI](https://github.com/drivendataorg/zamba/releases/new). The tag version must have a prefix `v`, e.g., `v2.0.1`.
+
+On publishing of the release, the [`release`](https://github.com/drivendataorg/zamba/blob/master/.github/workflows/release.yml) GitHub action workflow will be triggered. This workflow builds the package and publishes it to PyPI. You will be able to see the workflow status in the [Actions tab](https://github.com/drivendataorg/zamba/actions?query=workflow%3Arelease).
+
+## Documentation Website
+
+The documentation website is an mkdocs static site hosted on Netlify. The built website assets are first staged on the [`gh-pages` branch](https://github.com/drivendataorg/zamba/tree/gh-pages) and then deployed to Netlify automatically using GitHub Actions workflows.
+
+We use the [`mike` tool](https://github.com/jimporter/mike) to manage the documentation versions with the following conventions.
+- We keep the docs of the latest patch of a `<major>.<minor>` version, e.g., `v2.2.1` is keyed as `"v2.2"` and titled as `"v2.2.1"`.
+- The current stable version is tagged with the alias `"stable"` and has `"(stable)"` as part of the title.
+- The head of the `master` branch is keyed as `"~latest"` and titled as `"latest"`
+
+**To deploy the latest docs from the master branch, all you need to do is manually trigger the [`docs-latest` GitHub Actions workflow](https://github.com/drivendataorg/zamba/actions/workflows/docs-latest.yml).**
+
+**To manually deploy a previously released version, you will need to use `mike`. Follow the instructions in the following section.**
+
+### Manual deploy
+
+Note that `mike` needs to be run in the same direct as `mkdocs.yml`. To avoid changing directories all the time (since we keep it inside the `docs/` subdirectory), you can shadow the `mike` command with the following shell function:
+
+```bash
+# Put this in your .bash_profile or wherever you put aliases
+mike() {
+    if [[ -f mkdocs.yml ]]; then
+        command mike "$@"
+    else
+        (cd docs && command mike "$@")
+    fi
+}
+```
+
+The general steps of deploying docs for a specific version involve:
+
+1. Make sure your local `gh-pages` is up to date with GitHub, i.e., `git fetch origin && git checkout gh-pages && git pull origin gh-pages`
+2. Switch to whatever commit you're intending to deploy from.
+3. Run `make docs`. (This is necessary because of steps needed before running mkdocs things.)
+4. Run whatever `mike` command (see below). If you include the `--push` flag, it will also directly push your changes to GitHub. If you don't, it will only commit to your local `gh-pages` and you'll need to then push that branch to GitHub.
+5. Trigger the `docs-latest` GitHub actions workflow, which will deploy the `gh-pages` branch to Netlify.
+
+Staging the stable version will be something like this:
+
+```bash
+mike deploy v2.1 stable --title="v2.1.1 (stable)" --no-redirect --update-aliases
+```
+
+Staging an older version looks something like this:
+
+```bash
+mike deploy v1.1 --title="v1.1.5"
+```
+
+## Adding a new model
+
+Only models that were trained with `zamba` can be released.
+
+### Training a model with zamba
+
+To train a new model, specify the desired parameters in a config.yaml and then run
+```
+zamba train --config my_config.yaml
+```
+
+To retrain an existing model, it's easiest to work from one of the templates. You'll want to add any data and machine specific parameters.
+
+If you're retraining and not finetuning, remember to set `from_scratch: true`. This means starting with ImageNet weights for TimeDistributedEfficientNet and Ketics weights for SlowFast.
+
+For example, you might want to add the following to `train_config` in your model template of choice.
+
+```
+labels: my_labels.csv
+batch_size: 2
+num_workers: 3
+gpus: 1
+save_dir: my_save_dir
+skip_load_validation: true
+from_scratch: true
+```
+
+### Publishing a model
+
+Once your model is trained, copy the resulting directory to a folder in `s3://drivendata-client-zamba/data/results`. The directory should contain the following files:
+
+- model checkpoint file
+- `config.yaml` (configuration used for training)
+- `train_configuration.yaml` (automatically generated)
+- `predict_configuration.yaml` (automatically generated)
+- `hparams.yaml` (automatically generated)
+
+*Note: all files should be in the same level.*
+
+To publish an improved version of an existing model,
+- update the folder containing model weights in `WEIGHT_LOOKUP` in `zamba/models/config.py`
+
+If you are adding a new model,
+- add the model and weights folder as a new entry in the `WEIGHT_LOOKUP` dictionary
+- add the model name to `ModelEnum`
+- incorporate the new model into the test suite
+
+Then, run
+```
+make publish_models
+```
+
+This will generate a public file name for each model based on the config hash and upload the model weights to the three DrivenData public s3 buckets. This will generate a folder in `zamba/models/official_models/{your_name_name}` that contains the official config as well as reference yaml and json files. You should PR everything in this folder.
+
+Lastly, you need to update the template in `templates`. The template should contain all the same info as the model's `config.yaml`, plus placeholders for `data_dir` and `labels` in `train_config`, and `data_dir`, `filepaths`, and `checkpoint` in `predict_config`.
+
+### New model checklist
+
+- [ ] copy trained model folder to s3 with checkpoint, yaml, and json files
+- [ ] update private weights file in `WEIGHT_LOOKUP`
+- [ ] run `make publish_models` to generate or update configs in official models folder
+- [ ] update model template
+- [ ] PR template and subfolder in `official_models`
diff --git a/.github/codecov.yml b/.github/codecov.yml
@@ -0,0 +1,20 @@
+
+codecov:
+  require_ci_to_pass: yes
+
+coverage:
+  precision: 1
+  round: down
+  range: "50...100"
+  status:
+    project: # Coverage of whole project
+      default:
+        target: auto # Coverage target to pass; auto is base commit
+        threshold: 5% # Allow coverage to drop by this much vs. base and still pass
+    patch: # Coverage of lines in this change
+      default:
+        target: 50% # Coverage target to pass
+        threshold: 20% # Allow coverage to drop by this much vs. base and still pass
+
+comment:
+  layout: "diff,flags,tree"
diff --git a/.github/failed_build_issue_template.md b/.github/failed_build_issue_template.md
@@ -0,0 +1,5 @@
+---
+title: "Failed build on master branch ({{ env.WORKFLOW_NAME}} #{{ env.RUN_NUMBER }})"
+labels: bug
+---
+Workflow failed: [{{ env.WORKFLOW_NAME}} #{{ env.RUN_NUMBER }}](https://github.com/{{ env.REPOSITORY }}/actions/runs/{{ env.RUN_ID }})
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
diff --git a/.github/workflows/docs-latest.yml b/.github/workflows/docs-latest.yml
@@ -0,0 +1,62 @@
+name: docs-latest
+
+on:
+  push:
+    branches: [master, v2]
+  workflow_dispatch:
+
+jobs:
+  build:
+    name: Build 'latest' docs
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev/docs.txt
+          pip install -e .
+
+      - name: Build documentation
+        run: |
+          make docs
+
+      - name: Stage docs on gh-pages
+        working-directory: docs
+        run: |
+          git fetch origin gh-pages --depth=1
+          git config user.name github-actions[bot]
+          git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+          mike deploy --push ~latest --title=latest
+
+  deploy:
+    name: Deploy docs to Netlify
+    needs: build
+    runs-on: "ubuntu-latest"
+
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          ref: gh-pages
+
+      - name: Deploy docs to Netlify
+        uses: nwtgck/actions-netlify@v1.1
+        with:
+          publish-dir: "./"
+          production-deploy: true
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          deploy-message: "Deploy from GitHub Actions"
+          enable-pull-request-comment: false
+          enable-commit-comment: false
+          overwrites-pull-request-comment: false
+        env:
+          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+          NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
+        timeout-minutes: 1
diff --git a/.github/workflows/docs-preview.yml b/.github/workflows/docs-preview.yml
@@ -0,0 +1,43 @@
+name: docs-preview
+
+on:
+  pull_request:
+
+jobs:
+  build:
+    name: Build and deploy docs website preview
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev/docs.txt
+          pip install -e .
+
+      - name: Build documentation
+        run: |
+          make docs
+
+      - name: Deploy site preview to Netlify
+        uses: nwtgck/actions-netlify@v1.1
+        with:
+          publish-dir: "./docs/site"
+          production-deploy: false
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          deploy-message: "Deploy from GitHub Actions"
+          enable-pull-request-comment: true
+          enable-commit-comment: false
+          overwrites-pull-request-comment: true
+          alias: deploy-preview-${{ github.event.number }}
+        env:
+          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+          NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
+        timeout-minutes: 1
diff --git a/.github/workflows/publish-docs.yml b/.github/workflows/publish-docs.yml