Improved utilities for testing custom plugins

[jack-tutor]

describe the request

I am trying to write tests for a custom plugin. I'm running into issues similar to: #705 - with a couple of proposed solutions - hence making one semi-generic feature request here.

Request 1:

• Enable a mode that errors/raises a warning if a noqa comment is unused.
  I haven't actually been able to find too many requests for this feature - which I expect means I am just not finding them. Basically, I'm interested in the equivalent of this feature from mypy: https://mypy.readthedocs.io/en/stable/command_line.html?highlight=strict#cmdoption-mypy-warn-unused-ignores
  This would solve my testing issue as my current testing scheme is to just have a bunch of example files that answer the rules with the errors #noqa-ed. But, this only actually catches false positives, not false negatives. This is also a generally nice feature as it means your code will not get littered with empty lint checks. Perhaps the objection to this is that flake8 is a linting framework and does not implement any checks - I could see an argument either way here.

Request 2:

• For the python api, expose a more verbose pythonic output. Specifically, I'd like to know the line numbers of the non-ignored errors. I'd love to be able to run the api on a file and assert a certain error occurred on these 7 lines, another error occurred on these 3 lines, and no other errors occurred.

I'm trying to avoid either: diving into the internals of the flake8 python API, overriding the logging to get errors to tests, reparsing strings back into objects, or calling my custom linter directly as I'm interested in testing the interaction layer between my code and flake8.

Thanks!

[asottile]

part 1 has nothing to do with testing and is a duplicate of #603

the other part isn't happening -- if you need to do debugging you should use a debugger

[jack-tutor]

Thank you for linking me to #603 - I was sure other people would have had the request and could not find it.

I would clarify that I am not looking for a debugger with request 2 - I am asking for the Public Python API - defined here https://flake8.pycqa.org/en/latest/user/python-api.html to be more powerful. Though ill supported and legacy, the docs imply that future effort will be made to support this more fully. When that happens, I would hope that the Report object expose more information that is easier to use programatically, more information than just the total number of errors and summary statistics about specific violations. It's exposing a very limited API and the check_files call is logging the information I want to have programatically.

[asottile]

then that's a duplicate of #140 -- two for two :)