Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI Regression?] Re-enable flake8 #2501

Closed
webknjaz opened this issue Dec 21, 2020 · 13 comments
Closed

[CI Regression?] Re-enable flake8 #2501

webknjaz opened this issue Dec 21, 2020 · 13 comments
Labels
bug

Comments

@webknjaz
Copy link
Member

@jaraco it looks like after #2486 flake8 is always skipped:

$ tox -- --flake8 -vv -k FLAKE8
python develop-inst-noop: ~/src/github/pypa/setuptools
python installed: apipkg==1.5,appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,contextlib2==0.6.0.post1,coverage==5.3.1,distlib==0.3.1,docutils==0.16,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,flake8-2020==1.6.0,iniconfig==1.1.1,jaraco.apt==2.0,jaraco.classes==3.1.0,jaraco.collections==3.0.0,jaraco.context==3.0.0,jaraco.envs==2.0.0,jaraco.functools==3.0.1,jaraco.test==4.0.1,jaraco.text==3.2.0,mccabe==0.6.1,mock==4.0.3,more-itertools==8.6.0,mypy==0.790,mypy-extensions==0.4.3,packaging==20.8,path==15.0.1,path.py==12.5.0,pathspec==0.8.1,Paver==1.3.4,pip==20.2.4,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.2.1,pytest-black==0.3.12,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.7,pytest-mypy==0.8.0,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,pytz==2020.4,regex==2020.11.13,-e git+ssh://git@github.com/pypa/setuptools.git@ac2311014ab27409f1ca109101131fd7383cdc32#egg=setuptools,six==1.15.0,tempora==4.0.1,termcolor==1.1.0,toml==0.10.2,tox==3.20.1,tox-venv==0.4.0,typed-ast==1.4.1,typing-extensions==3.7.4.3,virtualenv==20.2.2,wheel==0.35.1,yg.lockfile==2.3,zc.lockfile==2.0
python run-test-pre: PYTHONHASHSEED='14316484'
python run-test: commands[0] | pytest --flake8 -vv -k FLAKE8
================================ test session starts ================================
platform linux -- Python 3.9.0, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- ~/src/github/pypa/setuptools/.tox/python/bin/python
cachedir: .tox/python/.pytest_cache
rootdir: ~/src/github/pypa/setuptools, configfile: pytest.ini
plugins: checkdocs-2.1.1, shutil-1.7.0, virtualenv-1.7.0, flake8-1.0.7, black-0.3.12, cov-2.10.1, mypy-0.8.0, jaraco.test-4.0.1
collected 805 items / 690 deselected / 2 skipped / 113 selected                     

bootstrap.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)        [  0%]
conftest.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)         [  1%]
easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)     [  2%]
[...snip...]
setuptools/installer.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8...) [ 22%]
setuptools/launch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 ch...) [ 23%]
setuptools/lib2to3_ex.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE...) [ 24%]
setuptools/monkey.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 ch...) [ 25%]
setuptools/msvc.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)  [ 26%]
setuptools/namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE...) [ 26%]
setuptools/package_index.py::FLAKE8 SKIPPED (file(s) previously passed FL...) [ 27%]
setuptools/py34compat.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE...) [ 28%]
[...snip...]
tools/ppc64le-patch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 ...) [ 99%]
tools/tox_pip.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)    [100%]

----------- coverage: platform linux, python 3.9.0-final-0 -----------
Name                                              Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------------
_distutils_hack/__init__.py                          66     43    35%   12-18, 28-33, 45-52, 63-64, 69-74, 77-88, 95-98, 105-106, 116, 120-123
_distutils_hack/override.py                           1      0   100%
[...snip...]
-------------------------------------------------------------------------------
TOTAL                                             17820  14002    21%


============================== short test summary info ==============================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
SKIPPED [115] .tox/python/lib/python3.9/site-packages/pytest_flake8.py:116: file(s) previously passed FLAKE8 checks
======================= 117 skipped, 690 deselected in 2.42s ========================
______________________________________ summary ______________________________________
  python: commands succeeded
  congratulations :)
@webknjaz
Copy link
Member Author

By the way, actually invoking .tox/python/bin/python -m flake8 directly runs for 1m 24s being extremely slow to recursively scan the dirs because of https://github.com/pypa/setuptools/pull/2486/files#r546877674.

@webknjaz
Copy link
Member Author

Excluding a few dirs makes flake8 finish in 16s (that's over 5x faster). Invoking flake8 revealed some other configuration problems. It makes me think that flake8 wasn't run in CI long before #2486.

@webknjaz webknjaz added the bug label Dec 21, 2020
webknjaz added a commit to webknjaz/setuptools that referenced this issue Dec 21, 2020
@webknjaz
Migrate to extend-exclude in flake8 config
f0e2c69 
This makes sure that flake8's defaults are in use. Specifically, it
excludes `.tox/` dir that is known to contain a lot of files in nested
folders that are not supposed to be linted.

Refs:
* pypa#2501 (comment)
* https://github.com/pypa/setuptools/pull/2486/files#r546877674
* https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-extend-exclude
* https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-exclude
@jaraco
Copy link
Member

jaraco commented Dec 21, 2020

It's working as intended for me:

setuptools main $ git clean -fdx
...
setuptools main $ tox -- -k flake8 -v -p no:cov
python create: /Users/jaraco/code/public/pypa/setuptools/.tox/python
python develop-inst: /Users/jaraco/code/public/pypa/setuptools
python installed: apipkg==1.5,appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,contextlib2==0.6.0.post1,coverage==5.3.1,distlib==0.3.1,docutils==0.16,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,flake8-2020==1.6.0,iniconfig==1.1.1,jaraco.apt==2.0,jaraco.classes==3.1.0,jaraco.collections==3.0.0,jaraco.context==3.0.0,jaraco.envs==2.0.0,jaraco.functools==3.0.1,jaraco.test==4.0.1,jaraco.text==3.2.0,mccabe==0.6.1,mock==4.0.3,more-itertools==8.6.0,mypy==0.790,mypy-extensions==0.4.3,packaging==20.8,path==15.0.1,path.py==12.5.0,pathspec==0.8.1,Paver==1.3.4,pip==20.2.4,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.2.1,pytest-black==0.3.12,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.7,pytest-mypy==0.8.0,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,pytz==2020.4,regex==2020.11.13,-e git+gh://pypa/setuptools@ac2311014ab27409f1ca109101131fd7383cdc32#egg=setuptools,six==1.15.0,tempora==4.0.1,termcolor==1.1.0,toml==0.10.2,tox==3.20.1,tox-venv==0.4.0,typed-ast==1.4.1,typing-extensions==3.7.4.3,virtualenv==20.2.2,wheel==0.35.1,yg.lockfile==2.3,zc.lockfile==2.0
python run-test-pre: PYTHONHASHSEED='35910917'
python run-test: commands[0] | pytest -k flake8 -v -p no:cov
========================================================== test session starts ==========================================================
platform darwin -- Python 3.9.0, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/jaraco/code/public/pypa/setuptools/.tox/python/bin/python
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/public/pypa/setuptools, configfile: pytest.ini
plugins: flake8-1.0.7, mypy-0.8.0, black-0.3.12, jaraco.test-4.0.1, shutil-1.7.0, checkdocs-2.1.1, virtualenv-1.7.0
collected 805 items / 690 deselected / 2 skipped / 113 selected                                                                         

bootstrap.py::FLAKE8 PASSED                                                                                                       [  0%]
conftest.py::FLAKE8 PASSED                                                                                                        [  1%]
easy_install.py::FLAKE8 PASSED                                                                                                    [  2%]
pavement.py::FLAKE8 PASSED                                                                                                        [  3%]
setup.py::FLAKE8 PASSED                                                                                                           [  4%]
docs/conf.py::FLAKE8 PASSED                                                                                                       [  5%]
pkg_resources/__init__.py::FLAKE8 PASSED                                                                                          [  6%]
pkg_resources/tests/__init__.py::FLAKE8 PASSED                                                                                    [  6%]
pkg_resources/tests/test_find_distributions.py::FLAKE8 PASSED                                                                     [  7%]
pkg_resources/tests/test_markers.py::FLAKE8 PASSED                                                                                [  8%]
pkg_resources/tests/test_pkg_resources.py::FLAKE8 PASSED                                                                          [  9%]
pkg_resources/tests/test_resources.py::FLAKE8 PASSED                                                                              [ 10%]
pkg_resources/tests/test_working_set.py::FLAKE8 PASSED                                                                            [ 11%]
setuptools/__init__.py::FLAKE8 PASSED                                                                                             [ 12%]
setuptools/_deprecation_warning.py::FLAKE8 PASSED                                                                                 [ 13%]
setuptools/_imp.py::FLAKE8 PASSED                                                                                                 [ 13%]
setuptools/archive_util.py::FLAKE8 PASSED                                                                                         [ 14%]
setuptools/build_meta.py::FLAKE8 PASSED                                                                                           [ 15%]
setuptools/config.py::FLAKE8 PASSED                                                                                               [ 16%]
setuptools/dep_util.py::FLAKE8 PASSED                                                                                             [ 17%]
setuptools/depends.py::FLAKE8 PASSED                                                                                              [ 18%]
setuptools/dist.py::FLAKE8 PASSED                                                                                                 [ 19%]
setuptools/errors.py::FLAKE8 PASSED                                                                                               [ 20%]
setuptools/extension.py::FLAKE8 PASSED                                                                                            [ 20%]
setuptools/glob.py::FLAKE8 PASSED                                                                                                 [ 21%]
setuptools/installer.py::FLAKE8 PASSED                                                                                            [ 22%]
setuptools/launch.py::FLAKE8 PASSED                                                                                               [ 23%]
setuptools/lib2to3_ex.py::FLAKE8 PASSED                                                                                           [ 24%]
setuptools/monkey.py::FLAKE8 PASSED                                                                                               [ 25%]
setuptools/msvc.py::FLAKE8 PASSED                                                                                                 [ 26%]
setuptools/namespaces.py::FLAKE8 PASSED                                                                                           [ 26%]
setuptools/package_index.py::FLAKE8 PASSED                                                                                        [ 27%]
setuptools/py34compat.py::FLAKE8 PASSED                                                                                           [ 28%]
setuptools/sandbox.py::FLAKE8 PASSED                                                                                              [ 29%]
setuptools/ssl_support.py::FLAKE8 PASSED                                                                                          [ 30%]
setuptools/unicode_utils.py::FLAKE8 PASSED                                                                                        [ 31%]
setuptools/version.py::FLAKE8 PASSED                                                                                              [ 32%]
setuptools/wheel.py::FLAKE8 PASSED                                                                                                [ 33%]
setuptools/windows_support.py::FLAKE8 PASSED                                                                                      [ 33%]
setuptools/command/__init__.py::FLAKE8 PASSED                                                                                     [ 34%]
setuptools/command/alias.py::FLAKE8 PASSED                                                                                        [ 35%]
setuptools/command/bdist_egg.py::FLAKE8 PASSED                                                                                    [ 36%]
setuptools/command/bdist_rpm.py::FLAKE8 PASSED                                                                                    [ 37%]
setuptools/command/bdist_wininst.py::FLAKE8 PASSED                                                                                [ 38%]
setuptools/command/build_clib.py::FLAKE8 PASSED                                                                                   [ 39%]
setuptools/command/build_ext.py::FLAKE8 PASSED                                                                                    [ 40%]
setuptools/command/build_py.py::FLAKE8 PASSED                                                                                     [ 40%]
setuptools/command/develop.py::FLAKE8 PASSED                                                                                      [ 41%]
setuptools/command/dist_info.py::FLAKE8 PASSED                                                                                    [ 42%]
setuptools/command/easy_install.py::FLAKE8 PASSED                                                                                 [ 43%]
setuptools/command/egg_info.py::FLAKE8 PASSED                                                                                     [ 44%]
setuptools/command/install.py::FLAKE8 PASSED                                                                                      [ 45%]
setuptools/command/install_egg_info.py::FLAKE8 PASSED                                                                             [ 46%]
setuptools/command/install_lib.py::FLAKE8 PASSED                                                                                  [ 46%]
setuptools/command/install_scripts.py::FLAKE8 PASSED                                                                              [ 47%]
setuptools/command/py36compat.py::FLAKE8 PASSED                                                                                   [ 48%]
setuptools/command/register.py::FLAKE8 PASSED                                                                                     [ 49%]
setuptools/command/rotate.py::FLAKE8 PASSED                                                                                       [ 50%]
setuptools/command/saveopts.py::FLAKE8 PASSED                                                                                     [ 51%]
setuptools/command/sdist.py::FLAKE8 PASSED                                                                                        [ 52%]
setuptools/command/setopt.py::FLAKE8 PASSED                                                                                       [ 53%]
setuptools/command/test.py::FLAKE8 PASSED                                                                                         [ 53%]
setuptools/command/upload.py::FLAKE8 PASSED                                                                                       [ 54%]
setuptools/command/upload_docs.py::FLAKE8 PASSED                                                                                  [ 55%]
setuptools/tests/__init__.py::FLAKE8 PASSED                                                                                       [ 56%]
setuptools/tests/contexts.py::FLAKE8 PASSED                                                                                       [ 57%]
setuptools/tests/environment.py::FLAKE8 PASSED                                                                                    [ 58%]
setuptools/tests/files.py::FLAKE8 PASSED                                                                                          [ 59%]
setuptools/tests/fixtures.py::FLAKE8 PASSED                                                                                       [ 60%]
setuptools/tests/namespaces.py::FLAKE8 PASSED                                                                                     [ 60%]
setuptools/tests/script-with-bom.py::FLAKE8 PASSED                                                                                [ 61%]
setuptools/tests/server.py::FLAKE8 PASSED                                                                                         [ 62%]
setuptools/tests/test_archive_util.py::FLAKE8 PASSED                                                                              [ 63%]
setuptools/tests/test_bdist_deprecations.py::FLAKE8 PASSED                                                                        [ 64%]
setuptools/tests/test_bdist_egg.py::FLAKE8 PASSED                                                                                 [ 65%]
setuptools/tests/test_build_clib.py::FLAKE8 PASSED                                                                                [ 66%]
setuptools/tests/test_build_ext.py::FLAKE8 PASSED                                                                                 [ 66%]
setuptools/tests/test_build_meta.py::FLAKE8 PASSED                                                                                [ 67%]
setuptools/tests/test_build_py.py::FLAKE8 PASSED                                                                                  [ 68%]
setuptools/tests/test_config.py::FLAKE8 PASSED                                                                                    [ 69%]
setuptools/tests/test_dep_util.py::FLAKE8 PASSED                                                                                  [ 70%]
setuptools/tests/test_depends.py::FLAKE8 PASSED                                                                                   [ 71%]
setuptools/tests/test_develop.py::FLAKE8 PASSED                                                                                   [ 72%]
setuptools/tests/test_dist.py::FLAKE8 PASSED                                                                                      [ 73%]
setuptools/tests/test_dist_info.py::FLAKE8 PASSED                                                                                 [ 73%]
setuptools/tests/test_distutils_adoption.py::FLAKE8 PASSED                                                                        [ 74%]
setuptools/tests/test_easy_install.py::FLAKE8 PASSED                                                                              [ 75%]
setuptools/tests/test_egg_info.py::FLAKE8 PASSED                                                                                  [ 76%]
setuptools/tests/test_extern.py::FLAKE8 PASSED                                                                                    [ 77%]
setuptools/tests/test_find_packages.py::FLAKE8 PASSED                                                                             [ 78%]
setuptools/tests/test_glob.py::FLAKE8 PASSED                                                                                      [ 79%]
setuptools/tests/test_install_scripts.py::FLAKE8 PASSED                                                                           [ 80%]
setuptools/tests/test_integration.py::FLAKE8 PASSED                                                                               [ 80%]
setuptools/tests/test_manifest.py::FLAKE8 PASSED                                                                                  [ 81%]
setuptools/tests/test_msvc.py::FLAKE8 PASSED                                                                                      [ 82%]
setuptools/tests/test_msvc14.py::FLAKE8 PASSED                                                                                    [ 83%]
setuptools/tests/test_namespaces.py::FLAKE8 PASSED                                                                                [ 84%]
setuptools/tests/test_packageindex.py::FLAKE8 PASSED                                                                              [ 85%]
setuptools/tests/test_register.py::FLAKE8 PASSED                                                                                  [ 86%]
setuptools/tests/test_sandbox.py::FLAKE8 PASSED                                                                                   [ 86%]
setuptools/tests/test_sdist.py::FLAKE8 PASSED                                                                                     [ 87%]
setuptools/tests/test_setopt.py::FLAKE8 PASSED                                                                                    [ 88%]
setuptools/tests/test_setuptools.py::FLAKE8 PASSED                                                                                [ 89%]
setuptools/tests/test_test.py::FLAKE8 PASSED                                                                                      [ 90%]
setuptools/tests/test_unicode_utils.py::FLAKE8 PASSED                                                                             [ 91%]
setuptools/tests/test_upload.py::FLAKE8 PASSED                                                                                    [ 92%]
setuptools/tests/test_upload_docs.py::FLAKE8 PASSED                                                                               [ 93%]
setuptools/tests/test_virtualenv.py::FLAKE8 PASSED                                                                                [ 93%]
setuptools/tests/test_wheel.py::FLAKE8 PASSED                                                                                     [ 94%]
setuptools/tests/test_windows_wrappers.py::FLAKE8 PASSED                                                                          [ 95%]
setuptools/tests/text.py::FLAKE8 PASSED                                                                                           [ 96%]
setuptools/tests/textwrap.py::FLAKE8 PASSED                                                                                       [ 97%]
tools/finalize.py::FLAKE8 PASSED                                                                                                  [ 98%]
tools/ppc64le-patch.py::FLAKE8 PASSED                                                                                             [ 99%]
tools/tox_pip.py::FLAKE8 PASSED                                                                                                   [100%]

======================================================== short test summary info ========================================================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
============================================ 115 passed, 2 skipped, 690 deselected in 12.79s ============================================
________________________________________________________________ summary ________________________________________________________________
  python: commands succeeded
  congratulations :)

setuptools main $ tox -- -k flake8 -v -p no:cov
python develop-inst-noop: /Users/jaraco/code/public/pypa/setuptools
python installed: apipkg==1.5,appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,contextlib2==0.6.0.post1,coverage==5.3.1,distlib==0.3.1,docutils==0.16,execnet==1.7.1,filelock==3.0.12,flake8==3.8.4,flake8-2020==1.6.0,iniconfig==1.1.1,jaraco.apt==2.0,jaraco.classes==3.1.0,jaraco.collections==3.0.0,jaraco.context==3.0.0,jaraco.envs==2.0.0,jaraco.functools==3.0.1,jaraco.test==4.0.1,jaraco.text==3.2.0,mccabe==0.6.1,mock==4.0.3,more-itertools==8.6.0,mypy==0.790,mypy-extensions==0.4.3,packaging==20.8,path==15.0.1,path.py==12.5.0,pathspec==0.8.1,Paver==1.3.4,pip==20.2.4,pluggy==0.13.1,py==1.10.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.2.1,pytest-black==0.3.12,pytest-checkdocs==2.1.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.7,pytest-mypy==0.8.0,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,pytz==2020.4,regex==2020.11.13,-e git+gh://pypa/setuptools@ac2311014ab27409f1ca109101131fd7383cdc32#egg=setuptools,six==1.15.0,tempora==4.0.1,termcolor==1.1.0,toml==0.10.2,tox==3.20.1,tox-venv==0.4.0,typed-ast==1.4.1,typing-extensions==3.7.4.3,virtualenv==20.2.2,wheel==0.35.1,yg.lockfile==2.3,zc.lockfile==2.0
python run-test-pre: PYTHONHASHSEED='2739366957'
python run-test: commands[0] | pytest -k flake8 -v -p no:cov
========================================================== test session starts ==========================================================
platform darwin -- Python 3.9.0, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/jaraco/code/public/pypa/setuptools/.tox/python/bin/python
cachedir: .tox/python/.pytest_cache
rootdir: /Users/jaraco/code/public/pypa/setuptools, configfile: pytest.ini
plugins: flake8-1.0.7, mypy-0.8.0, black-0.3.12, jaraco.test-4.0.1, shutil-1.7.0, checkdocs-2.1.1, virtualenv-1.7.0
collected 805 items / 690 deselected / 2 skipped / 113 selected                                                                         

bootstrap.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                            [  0%]
conftest.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                             [  1%]
easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                         [  2%]
pavement.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                             [  3%]
setup.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                                [  4%]
docs/conf.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                            [  5%]
pkg_resources/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                               [  6%]
pkg_resources/tests/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [  6%]
pkg_resources/tests/test_find_distributions.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                          [  7%]
pkg_resources/tests/test_markers.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [  8%]
pkg_resources/tests/test_pkg_resources.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                               [  9%]
pkg_resources/tests/test_resources.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 10%]
pkg_resources/tests/test_working_set.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                 [ 11%]
setuptools/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                  [ 12%]
setuptools/_deprecation_warning.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 13%]
setuptools/_imp.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 13%]
setuptools/archive_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                              [ 14%]
setuptools/build_meta.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 15%]
setuptools/config.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 16%]
setuptools/dep_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                  [ 17%]
setuptools/depends.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                   [ 18%]
setuptools/dist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 19%]
setuptools/errors.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 20%]
setuptools/extension.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                 [ 20%]
setuptools/glob.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 21%]
setuptools/installer.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                 [ 22%]
setuptools/launch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 23%]
setuptools/lib2to3_ex.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 24%]
setuptools/monkey.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                    [ 25%]
setuptools/msvc.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                      [ 26%]
setuptools/namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 26%]
setuptools/package_index.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 27%]
setuptools/py34compat.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 28%]
setuptools/sandbox.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                   [ 29%]
setuptools/ssl_support.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                               [ 30%]
setuptools/unicode_utils.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 31%]
setuptools/version.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                   [ 32%]
setuptools/wheel.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                     [ 33%]
setuptools/windows_support.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 33%]
setuptools/command/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 34%]
setuptools/command/alias.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 35%]
setuptools/command/bdist_egg.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 36%]
setuptools/command/bdist_rpm.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 37%]
setuptools/command/bdist_wininst.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 38%]
setuptools/command/build_clib.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 39%]
setuptools/command/build_ext.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 40%]
setuptools/command/build_py.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 40%]
setuptools/command/develop.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 41%]
setuptools/command/dist_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 42%]
setuptools/command/easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 43%]
setuptools/command/egg_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 44%]
setuptools/command/install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 45%]
setuptools/command/install_egg_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                  [ 46%]
setuptools/command/install_lib.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 46%]
setuptools/command/install_scripts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 47%]
setuptools/command/py36compat.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 48%]
setuptools/command/register.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 49%]
setuptools/command/rotate.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 50%]
setuptools/command/saveopts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 51%]
setuptools/command/sdist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                             [ 52%]
setuptools/command/setopt.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 53%]
setuptools/command/test.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                              [ 53%]
setuptools/command/upload.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 54%]
setuptools/command/upload_docs.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 55%]
setuptools/tests/__init__.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 56%]
setuptools/tests/contexts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 57%]
setuptools/tests/environment.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 58%]
setuptools/tests/files.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                               [ 59%]
setuptools/tests/fixtures.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 60%]
setuptools/tests/namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 60%]
setuptools/tests/script-with-bom.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 61%]
setuptools/tests/server.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                              [ 62%]
setuptools/tests/test_archive_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 63%]
setuptools/tests/test_bdist_deprecations.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                             [ 64%]
setuptools/tests/test_bdist_egg.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 65%]
setuptools/tests/test_build_clib.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 66%]
setuptools/tests/test_build_ext.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 66%]
setuptools/tests/test_build_meta.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 67%]
setuptools/tests/test_build_py.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 68%]
setuptools/tests/test_config.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 69%]
setuptools/tests/test_dep_util.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 70%]
setuptools/tests/test_depends.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 71%]
setuptools/tests/test_develop.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 72%]
setuptools/tests/test_dist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 73%]
setuptools/tests/test_dist_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                      [ 73%]
setuptools/tests/test_distutils_adoption.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                             [ 74%]
setuptools/tests/test_easy_install.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 75%]
setuptools/tests/test_egg_info.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 76%]
setuptools/tests/test_extern.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 77%]
setuptools/tests/test_find_packages.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                  [ 78%]
setuptools/tests/test_glob.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 79%]
setuptools/tests/test_install_scripts.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                [ 80%]
setuptools/tests/test_integration.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                    [ 80%]
setuptools/tests/test_manifest.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 81%]
setuptools/tests/test_msvc.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 82%]
setuptools/tests/test_msvc14.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 83%]
setuptools/tests/test_namespaces.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 84%]
setuptools/tests/test_packageindex.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                   [ 85%]
setuptools/tests/test_register.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                       [ 86%]
setuptools/tests/test_sandbox.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                        [ 86%]
setuptools/tests/test_sdist.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 87%]
setuptools/tests/test_setopt.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 88%]
setuptools/tests/test_setuptools.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 89%]
setuptools/tests/test_test.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                           [ 90%]
setuptools/tests/test_unicode_utils.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                  [ 91%]
setuptools/tests/test_upload.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                         [ 92%]
setuptools/tests/test_upload_docs.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                    [ 93%]
setuptools/tests/test_virtualenv.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                     [ 93%]
setuptools/tests/test_wheel.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                          [ 94%]
setuptools/tests/test_windows_wrappers.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                               [ 95%]
setuptools/tests/text.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                [ 96%]
setuptools/tests/textwrap.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                            [ 97%]
tools/finalize.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                       [ 98%]
tools/ppc64le-patch.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                  [ 99%]
tools/tox_pip.py::FLAKE8 SKIPPED (file(s) previously passed FLAKE8 checks)                                                        [100%]

======================================================== short test summary info ========================================================
SKIPPED [2] setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
SKIPPED [115] .tox/python/lib/python3.9/site-packages/pytest_flake8.py:116: file(s) previously passed FLAKE8 checks
================================================= 117 skipped, 690 deselected in 1.79s ==================================================
________________________________________________________________ summary ________________________________________________________________
  python: commands succeeded
  congratulations :)

Note, you don't need to pass --flake8. With the inclusion of jaraco.test and the configuration in pyproject.toml, --flake8 will be implied if the pytest flake8 plugin is present and enabled (and not if not). By default, the plugin is present and enabled, but can be disabled with -p no:flake8 or uninstalling it.

Note that I also saw the skips on the second run. That's working as intended. The flake8 tests don't need to be run a second time if the files haven't changed since they were last checked.

Also, you can see the tests ran in 13s for me, and 1.8s for the second attempt.

I did disable coverage for these runs to avoid the noisy (buggy) output that coverage generates.

@webknjaz
Copy link
Member Author

Note that I also saw the skips on the second run. That's working as intended. The flake8 tests don't need to be run a second time if the files haven't changed since they were last checked.

It's interesting that I didn't see any PASSED lines and that's what I'm confused about. Still, I've found a number of misconfigured settings in the configs. Also, I think that flake8 should run w/o such problems separately because there's no other clean way of seeing what's happening when you run it.

@webknjaz
Copy link
Member Author

I've made an intentional syntax error in bootstrap.py but that fails on the collection step of pytest not even getting to flake8. I guess that's fine...

@jaraco
Copy link
Member

jaraco commented Dec 21, 2020

If you run git clean -fdx first, that should clear out the cache and I'd expect you'll get passed lines.

Agreed, it would be nice for flake8 to run naturally, though I'm okay with that usage being broken as long as the test suite is able to enforce the flake8 checks during the test run.

I believe the use of extend-ignore (jaraco/skeleton#28) should address that concern, as you describe.

@webknjaz
Copy link
Member Author

I've verified that making style violations adds the failed entries and fixing that produces PASSED. I didn't realize there's some cache. Anyway, it's quite confusing to see linters among regular tests.

I believe the use of extend-ignore (jaraco/skeleton#28) should address that concern, as you describe.

Almost. Also, there's extend-exclude that still results in some violations showing up (specifically under setuptools/_distutils/).

@webknjaz
Copy link
Member Author

FWIW I'll probably send a few PRs later to address what I've noticed.

@jaraco
Copy link
Member

jaraco commented Dec 21, 2020

FWIW I'll probably send a few PRs later to address what I've noticed.

Sounds good. For this first round, I'm aiming toward simplicity and conformance with the skeleton without unduly compromising the project's expectations. I'd like to let these changes stabilize before tweaking them further. Feel free to send PRs, but I may choose to sit on them for some time, especially if they're not critical changes.

@webknjaz
Copy link
Member Author

skeleton PR: jaraco/skeleton#33

webknjaz added a commit to webknjaz/setuptools that referenced this issue Dec 26, 2020
@webknjaz
Migrate to extend-exclude in flake8 config
c0c9cfd 
This makes sure that flake8's defaults are in use. Specifically, it
excludes `.tox/` dir that is known to contain a lot of files in nested
folders that are not supposed to be linted.

Refs:
* pypa#2501 (comment)
* https://github.com/pypa/setuptools/pull/2486/files#r546877674
* https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-extend-exclude
* https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-exclude
@webknjaz webknjaz mentioned this issue Dec 26, 2020
Merged
2 tasks
@webknjaz
Copy link
Member Author

setuptools PR: #2508

@webknjaz webknjaz mentioned this issue Dec 26, 2020
Merged
2 tasks
@webknjaz
Copy link
Member Author

Invalid config entries PR: #2509.

webknjaz added a commit to webknjaz/setuptools that referenced this issue Dec 26, 2020
@webknjaz
Use extend-ignore in flake8 config
33d09d1 
This option allows adding extra ignored rules to the default list
instead of replacing it.

The default exclusions are: E121, E123, E126, E226, E24, E704,
W503 and W504.

Refs:
* https://github.com/pypa/setuptools/pull/2486/files#r541943356
* https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-extend-ignore
* https://flake8.pycqa.org/en/latest/user/options.html#cmdoption-flake8-ignore
* pypa#2501
* https://github.com//skeleton/issues/28
@webknjaz webknjaz mentioned this issue Dec 26, 2020
Merged
2 tasks
@webknjaz
Copy link
Member Author

webknjaz commented Feb 4, 2021

It looks like this issue has been addressed.

@webknjaz webknjaz closed this as completed Feb 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jaraco @webknjaz