Adds black and pre-commit hooks (#239)

* Moved /ewatercycle to /src/ewatercycle + adjust config to match Refs #228 * Add black to dev deps Replaces yapf as formatter * Configure black to also format notebooks * Added black and friends as pre-commit hooks and to ci * Files changes by `pre-commit run --all-files` command * Use nbqa-black to format notebooks instead of black As black replaces " with ', which makes invalid JSON * On release run pre-commit on all files * Disable flake8 until its errors are fixed * Added flake8 plugins Selection from https://towardsdatascience.com/static-code-analysis-for-python-bdce10b8d287 * Make isort, pylint and flake8 compatible with black See https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html * Use tabs in makefile * Replace prospector with pre-commit * Prefer `@pytest.fixture` over `@pytest.fixture()` * Added pylint and bunch of flake8 plugins to pre commit config Using YAML anchor + alias for nb * Move config from arg to file + pytest-fixture config inverted * Disable pylint and fix flake8 errors first * Ignore docstring checks in tests/ + Move google doc style to where pydocstyle expects it * Fix flake8 errors * Reduce flake8 warnings * Fix doc generator + Fix bunch of lint warnings * More of the same * Fixing lint errors in grdc Fixes #241 * Make flake8 just print errors instead of dying * Fix line lengths in ewatercycle.models * Fix line lengths in config * Fix line lengths in ewatercycle.parameter_sets * fix line lengths in ewatercycle.forcing * Fix remaining line lengths in ewatercycle package * noqa instead of pylint-specific * Fix line lengths in tests * fix error message * Force showing of warnings of nbqa-flake8 * Update .pre-commit-config.yaml Co-authored-by: Peter Kalverla <peter.kalverla@gmx.com> Co-authored-by: Peter Kalverla <peter.kalverla@gmx.com>
eWaterCycle · Aug 20, 2021 · c794ef7 · c794ef7
1 parent 6a4fe28
commit c794ef7
Show file tree

Hide file tree

Showing 69 changed files with 2,306 additions and 1,964 deletions.
diff --git a/.editorconfig b/.editorconfig
@@ -25,3 +25,6 @@ indent_size = 2
 
 [*.{md,Rmd}]
 trim_trailing_whitespace = false
+
+[Makefile]
+indent_style = tab
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -1,13 +1,10 @@
-# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
-# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
-
 name: Python package
 
 on:
   push:
     branches: [main]
   pull_request:
-      types: [opened, synchronize, reopened]
+    types: [opened, synchronize, reopened]
 
 jobs:
   build:
@@ -19,18 +16,18 @@ jobs:
       fail-fast: false
     name: Run tests in conda environment ${{ matrix.python-version }}
     steps:
-    - uses: actions/checkout@v2
-    - uses: conda-incubator/setup-miniconda@v2
-      with:
-        activate-environment: ewatercycle
-        environment-file: environment.yml
-        python-version: ${{ matrix.python-version }}
-        miniconda-version: "latest"
-        channels: conda-forge
-    - name: Install dependencies
-      shell: bash -l {0}
-      run: |
-        pip3 install -e .[dev]
-    - name: Test with pytest
-      run: pytest
-      shell: bash -l {0}
+      - uses: actions/checkout@v2
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          activate-environment: ewatercycle
+          environment-file: environment.yml
+          python-version: ${{ matrix.python-version }}
+          miniconda-version: "latest"
+          channels: conda-forge
+      - name: Install dependencies
+        shell: bash -l {0}
+        run: |
+          pip3 install -e .[dev]
+      - name: Test with pytest
+        run: pytest
+        shell: bash -l {0}
diff --git a/.github/workflows/sonar.yml b/.github/workflows/sonar.yml
@@ -4,34 +4,38 @@ on:
   push:
     branches: [main]
   pull_request:
-      types: [opened, synchronize, reopened]
+    types: [opened, synchronize, reopened]
 
 jobs:
   sonarcloud:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
-      with:
-        fetch-depth: 0
-    - name: Set up Python
-      uses: conda-incubator/setup-miniconda@v2
-      with:
-        activate-environment: ewatercycle
-        environment-file: environment.yml
-        python-version: 3.8
-        miniconda-version: "latest"
-        channels: conda-forge
-    - name: Install dependencies
-      shell: bash -l {0}
-      run: |
-        pip3 install -e .[dev]
-    - name: Tests with coverage
-      run: pytest --cov --cov-report term --cov-report xml --junitxml=xunit-result.xml
-      shell: bash -l {0}
-    - name: Correct coverage paths
-      run: sed -i "s+$PWD/++g" coverage.xml
-    - name: SonarCloud Scan
-      uses: sonarsource/sonarcloud-github-action@master
-      env:
-        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+      - name: Set up Python
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          activate-environment: ewatercycle
+          environment-file: environment.yml
+          python-version: 3.8
+          miniconda-version: "latest"
+          channels: conda-forge
+      - name: Install dependencies
+        shell: bash -l {0}
+        run: |
+          pip3 install -e .[dev]
+      - name: Run pre commit hooks like linters and black formatter
+        uses: pre-commit/action@v2.0.3
+      - name: Tests with coverage
+        run: |
+          pytest --cov --cov-report term --cov-report xml \
+            --junitxml=xunit-result.xml
+        shell: bash -l {0}
+      - name: Correct coverage paths
+        run: sed -i "s+$PWD/++g" coverage.xml
+      - name: SonarCloud Scan
+        uses: sonarsource/sonarcloud-github-action@master
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,98 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.4.0
+    hooks:
+      - id: check-added-large-files
+      - id: check-ast
+      - id: check-case-conflict
+      - id: check-merge-conflict
+      - id: debug-statements
+      - id: end-of-file-fixer
+      - id: trailing-whitespace
+        args: [--markdown-linebreak-ext=md]
+  - repo: https://github.com/adrienverge/yamllint
+    rev: "v1.26.0"
+    hooks:
+      - id: yamllint
+  - repo: https://github.com/asottile/setup-cfg-fmt
+    rev: v1.17.0
+    hooks:
+      - id: setup-cfg-fmt
+  - repo: https://github.com/psf/black
+    rev: 21.7b0
+    hooks:
+      - id: black
+  - repo: https://github.com/PyCQA/isort
+    rev: "5.9.3"
+    hooks:
+      - id: isort
+  # TODO renable when errors are fixed/ignored
+  # - repo: https://github.com/pycqa/pylint
+  #   rev: "v2.9.6"
+  #   hooks:
+  #     - id: pylint
+  # TODO renable when errors are fixed/ignored
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: "3.9.2"
+    hooks:
+      - id: flake8
+        additional_dependencies:
+          &fd [
+            flake8-annotations-complexity,
+            flake8-bandit,
+            flake8-blind-except,
+            flake8-bugbear,
+            flake8-builtins,
+            flake8-cognitive-complexity,
+            flake8-comprehensions,
+            flake8-docstrings,
+            flake8-eradicate,
+            flake8-executable,
+            flake8-expression-complexity,
+            flake8-if-expr,
+            flake8-implicit-str-concat,
+            flake8-logging-format,
+            flake8-pathlib,
+            flake8-print,
+            flake8-pytest,
+            flake8-pytest-style,
+            # flake8-quotes,  # conflicts with blacks double quote preference
+            flake8-raise,
+            flake8-return,
+            flake8-typing-imports,
+            flake8-variables-names,
+            flake8==3.9.2,
+            pandas-vet,
+            pep8-naming,
+            # wemake-python-styleguide,  # conflicts with black
+            yesqa,
+          ]
+        verbose: true
+        args: &fa [--statistics, --exit-zero]
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v0.910
+    hooks:
+      - id: mypy
+        additional_dependencies: [types-python-dateutil]
+  - repo: https://github.com/nbQA-dev/nbQA
+    rev: 1.1.0
+    hooks:
+      - id: nbqa-black
+        # Match version of black used for .py and .ipynb
+        additional_dependencies: [black==21.7b0]
+      - id: nbqa-isort
+        additional_dependencies: [isort==5.9.3]
+      - id: nbqa-mypy
+        additional_dependencies: [mypy==0.910]
+      # TODO renable when errors are fixed/ignored
+      - id: nbqa-flake8
+        additional_dependencies: *fd
+        args: *fa
+        verbose: true
+      # TODO renable when errors are fixed/ignored
+      # - id: nbqa-pylint
+      #   additional_dependencies: [pylint==2.9.6]
+  - repo: https://github.com/regebro/pyroma
+    rev: "3.2"
+    hooks:
+      - id: pyroma
diff --git a/.prospector.yml b/.prospector.yml
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -55,6 +55,7 @@ The sections below outline the steps in each case.
     and [here](https://help.github.com/articles/syncing-a-fork/));
 4. install the package in editable mode and its dependencies with
     `pip3 install -e .[dev]`;
+4. make sure pre commit hook is installed by running `pre-commit install`, causes linting and formatting to be applied during commit;
 5. make sure the existing tests still work by running `pytest`;
 6. make sure the existing documentation can still by generated without
     warnings by running `cd docs && make html`. [Pandoc](https://pandoc.org/) is required to generate docs, it can be installed with ``conda install -c conda-forge pandoc`` ;
@@ -81,24 +82,25 @@ This section is for maintainers of the package.
 2. Determine what new version (major, minor or patch) to use. Package uses `semantic versioning <https://semver.org>`_.
 3. Run ``bump2version <major|minor|patch>`` to update version in package files.
 4. Update CHANGELOG.md with changes between current and new version.
-5. Commit & push changes to GitHub.
-6. Wait for [GitHub
+5. Make sure pre-commit hooks are green for all files by running ``pre-commit run --all-files``.
+6. Commit & push changes to GitHub.
+7. Wait for [GitHub
     actions](https://github.com/eWaterCycle/ewatercycle/actions?query=branch%3Amain+)
     to be completed and green.
 
-7. Create a [GitHub release](https://github.com/eWaterCycle/ewatercycle/releases/new)
+8. Create a [GitHub release](https://github.com/eWaterCycle/ewatercycle/releases/new)
 
     - Use version as title and tag version.
     - As description use intro text from README.md (to give context to
         Zenodo record) and changes from CHANGELOG.md
 
-8. Create a PyPI release.
+9. Create a PyPI release.
 
     1. Create distribution archives with `python3 -m build`.
     2. Upload archives to PyPI with `twine upload dist/*` (use your
         personal PyPI account).
 
-9. Verify
+10. Verify
 
     1. Has [new Zenodo
         record](https://zenodo.org/search?page=1&size=20&q=ewatercycle)
@@ -109,4 +111,4 @@ This section is for maintainers of the package.
     3. Can new version be installed with pip using
         `pip3 install ewatercycle==<new version>`?
 
-10. Celebrate
+11. Celebrate
diff --git a/LICENSE b/LICENSE
@@ -14,4 +14,3 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-