py36+ tests are definition ordered

[asottile]

Resolves #5196
See #7808

[asottile]

an interesting side-effect of this is base classes appear after the subclass which is perhaps a little strange -- I can see if I can adjust that behaviour if it's undesirable (I would expect pytest to run base-class tests first, though it makes sense from an mro() perspective the way it is currently implemented) 🤔

[bluetech]

So IIUC the previous behavior was to execute tests in the order in which they physically appear (the file path and the line number within the file). So tests from base classes would execute before or after depending on where the base class is in the file (nice IMO) or how its filename sorts (doesn't make sense IMO).

Certainly definition order makes more sense semantically, but I do indeed think that tests from base classes should run first, that's how I expect it anyway. I was going to say "just traverse the MRO in reverse" but since the code pokes at __dict__s directly that would be tricky.

[asottile]

I don't think the mro thing would be that complicated -- I'll see what I can do (I agree the base classes should run first)

[bluetech]

Looking at collection timings for pandas (182330 tests), this sort call features prominently. Particularly the slowdown comes from sort -> sort_key -> PyObjMixin.reportinfo() -> getfslineno(obj) -> findsource(obj) (only for objs without __code__, mostly test classes) -> inspect.findsource(obj). And apparently since Python 3.9, inspect.findsource() uses ast instead of regexes to find the line number of classes, which I suspect makes it even slower (didn't verify this).

Anyway, if we can get this PR to work it will be nice (maybe I'll try it myself).

[asottile]

ah yeah I haven't had time to come back to this -- I think all that needs to happen is to consider the mro when ordering tests -- not sure when I'll have some time to look at it so feel free to carry the patch!