fail_under in .coveragerc does not raise error django 3.0.4

[prausinski]

HI!
Describe the bug
I configure the coverage.py through .coveragerc file (see below). I can see that coverage does the things I configure in coveragerc accordingly despite fail_under parameter. When defining the fail_under = 99.0 (both int and float does not work) parameter the coverage report command does not exit with a "fail" or something. Furthermore, using the option in the command coverage report --fail-under=99 does not raise a failure or something.

To Reproduce
Unfortunately, I cannot share the code through a repository or something. The thing is that the commands which I'm running do work the way they should do. However, I want the 'coverage report' respectively the coverage report --fail-under=99 command to fail or exit with failures linked to the threshold via fail_under = 99.0 (that I can use for Jenkins Pipeline later).
However, it is a basic Django 3.0.4 project with no extra magic using Django REST Framework 3.11. I did my first steps with django-nose in this project but then realized that django-nose is not properly supported anymore. So I decided to switch to "pure coverage" (this is why in pip freeze the django-nose is still listed)

Structure:
./app
/app/customer_db
/app/core
/app/api
/app/user
./manage.py
./coveragerc

1. What code are you running? Give us a specific commit of a specific repo that we can check out.

Commands:

1. coverage run manage.py test

OUTPUT:
Ran 166 tests in 7.632s

OK
Destroying test database for alias 'default'...

2. coverage report
   OUTPUT:

Name                                                                  Stmts   Miss Branch BrPart    Cover   Missing
-------------------------------------------------------------------------------------------------------------------
app/customer_db/validators.py                                            45     21     16      2    49.2%   32->33, 33, 40->50, 50-75, 79-90
app/user/authentication.py                                               34     12      6      1    57.5%   34->35, 35-36, 49-61
app/user/migrations/0002_groups.py                                       24      2      6      1    83.3%   14->28, 28-29
app/customer_db/models.py                                               147     14     16      4    89.0%   36, 57, 87->88, 88, 109, 132, 135->136, 136, 179->180, 180, 190, 214, 224, 275->276, 276, 295, 318, 325


etc. etc. etc.


-------------------------------------------------------------------------------------------------------------------
TOTAL                                                                  1214     79    170     18    92.0%

14 empty files skipped.

python --version:
Python 3.6.7

pip freeze:

asgiref==3.2.5
certifi==2019.11.28
click==7.1.1
coverage==5.1
Django==3.0.4
django-nose==1.4.6
django-simple-history==2.8.0
djangorestframework==3.11.0
flake8==3.6.0
fuzzywuzzy==0.18.0
geographiclib==1.50
geopy==1.21.0
gunicorn==20.0.4
joblib==0.14.1
kph==0.4.1
mccabe==0.6.1
nltk==3.5
nose==1.3.7
psycopg2==2.7.7
psycopg2-binary==2.8.4
pycodestyle==2.4.0
pyflakes==2.0.0
python-Levenshtein==0.12.0
pytz==2019.3
regex==2020.4.4
sentry-sdk==0.14.2
six==1.14.0
sqlparse==0.3.1
tqdm==4.45.0
urllib3==1.25.8

In .coveragerc:

[run]
branch = true
include =   ./app/api/*
            ./app/core/*
            ./app/customer_db/*
            ./app/user/*
[report]
fail_under = 99.0
omit = */tests/*
precision = 1
show_missing = true
skip_empty = true
sort = Cover
ignore_errors = true

coverage debug sys:

                        version: 5.1
                       coverage: User/dev/git_projects/mdm/mdm-backend/venv/lib/python3.6/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                   configs_read: User/dev/git_projects/mdm/mdm-backend/app/.coveragerc
                    config_file: User/dev/git_projects/mdm/mdm-backend/app/.coveragerc
                config_contents: '# coverage run settings\n[run]\nbranch = true\ninclude =   ./app/api/*\n            ./app/core/*\n            ./app/customer_db/*\n            ./app/user/*\n# coverage report settings\n[report]\nfail_under = 99.0\nomit = */tests/*\nprecision = 1\nshow_missing = true\nskip_empty = true\nsort = Cover\nignore_errors = true'
                      data_file: -none-
                         python: 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 00:28:22) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
                       platform: Darwin-19.3.0-x86_64-i386-64bit
                 implementation: CPython
                     executable: User/dev/git_projects/mdm/mdm-backend/venv/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 15778
                            cwd: User/dev/git_projects/mdm/mdm-backend/app
                           path: User/dev/git_projects/mdm/mdm-backend/venv/bin
                                 /Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip
                                 /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6
                                 /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload
                                 User/dev/git_projects/mdm/mdm-backend/venv/lib/python3.6/site-packages
                    environment: __PYVENV_LAUNCHER__ = User/dev/git_projects/mdm/mdm-backend/venv/bin/python
                   command_line: User/dev/git_projects/mdm/mdm-backend/venv/bin/coverage debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.22.0
             sqlite3_temp_store: 0
        sqlite3_compile_options: COMPILER=gcc-4.2.1 (Apple Inc. build 5666) (dot 3)
                                 ENABLE_FTS3
                                 ENABLE_FTS3_PARENTHESIS
                                 ENABLE_FTS4
                                 ENABLE_FTS5
                                 ENABLE_JSON1
                                 ENABLE_RTREE
                                 THREADSAFE=1

Expected behavior
When running coverage report after printing the list the command should according to the threshold defined through fail_under parameter in .coveragerc as well as in included in the command like coverage report --fail-under=99

[nedbat]

It's not clear to me how you are running the coverage commands. --fail-under can be mysterious because it's really quiet: it just affects the status code of the command. (See #977 for a change to show a message when it fails.)

Depending on how you are running the commands, you might not notice that it is returning a failure code.

[prausinski]

Hi Ned, thanks for your reply. Was not aware of the "mysterious silence". I now fetch the status code of the command and go on with it. Thank you.