Disable timing-out when a debugging session is in progress

[Mattwmaster58]

Fixes #32

This improved functionality is implemented by checking if pydevd.py appears in any of the stack frames. If it does, we assume that a debugging session is active. See discussion for evolution of detection.

Also updated:

• py.test -> pytest, py.test is apparently deprecated
• README.rst to reflect new functionality

[flub]

Thanks for the PR!
Could you explain what the pydevd.py heuristic is for? The only thing I could find was maybe https://github.com/fabioz/PyDev.Debugger/blob/master/pydevd.py ?
I'm curious how this debugger invokes tests, in particular your change seems to only check for the debugger presence at the start of the test. So I'm trying to understand the situations affected here.
I also wonder how much this perhaps should be turned around? Should the debugger be calling pytest hooks instead? I'm just widely brainstorming here, this could also be a terrible idea.
My last observation for now is that this doesn't come with any test... we should try and see if there's a reasonable way to test this.

[Mattwmaster58]

Could you explain what the pydevd.py heuristic is for?

Sure. pydevd.py IIUC is a debugging 'backend' used by PyDev, PyCharm and VSCode Python (as well as other's I'm sure). I can't speak for PyDev and VSCode, but PyCharm integrates its own pytest runner which I find extremely convenient to use.

So I'm trying to understand the situations affected here.

As far as I can tell, this would only effect things being debugged. Or if pydevd itself is being run but isn't debugging a session. I'm not quite sure there would ever be a scenario where 'pydevd.py' would be in the stack frames without a debugging session active.

I'm curious how this debugger invokes tests, in particular your change seems to only check for the debugger presence at the start of the test. I also wonder how much this perhaps should be turned around? Should the debugger be calling pytest hooks instead? I'm just widely brainstorming here, this could also be a terrible idea.

Initially, I thought it was the case that pydevd.py only allowed for debugging when initialized at the process startup. A quick google search reveals that this is incorrect, and that it is possible to attach to a running process. So I think it would be best to integrate this in the same place where the we check if SUPPRESS_TIMEOUT is True.

My last observation for now is that this doesn't come with any test... we should try and see if there's a reasonable way to test this.

Maybe, but I'm not quite sure how we'd test this. Maybe just calling a test with ridiculously small timeout that sleeps for 1s or whatever and assert that it does in fact take 1s.

[Mattwmaster58]

Sorry about all the formatting commits, couldn't get pre-commit setup until I modified the pre commit config to not rely on python2 (also included in this PR).

[flub]

I still find your suggested heuristic pretty scary to use, it feels super ad-hoc. Would this not work if you check sys.gettrace() is None for example? That's already much more a standard python interface. I feel like this also warrants careful reading of PEP553 to glean any knowledge.

But I'd also like to get back to the other point that perhaps the runner should be calling the pytest_enter_pdb hook. How does your debugger actually invoke pytest while the debugger is enabled? Does it call pytest.main()? Is the debugger already set by then, and how is it set?)

[Mattwmaster58]

You're right, and there's a performance hit too — around 50ms for my machine. I read over PEP553 and the PyDevD source code. to see if there was a better way to do this. I initially just threw this together thinking it would be the best way to do it based on . . . another source 😉 . My new approach should target all debuggers across 2.7-3.8 as long as they use a similar approach to PyDevD (specifically, utilizing sys.breakpointhook or builtins.breakpoint)

My strategy is to check

• sys.breakpointhook
• builtins.breakpoint (__builtin__ if Py 2)
• sys.__breakpointhook__

and see if the module the function belongs to is in the std lib — if not, I assume that it has been hooked and thus a debugging session is active. This would almost makes sense to check at startup, however that wouldn't cover the case of attaching to a process.

As for performance — the cost is around 2us/call on my machine.

WRT to using pytest_enter_pbd: IIUC, that will drop you into a pbd debugging session on test failure — this wouldn't even be workable for solutions (eg PyCharm) that provide there own debugging interface (again IIUC).

[flub]

There is however a difference between a breakpoint hook being installed and actually being in an active debugging session isn't there? I think it's possible for someone having installed a breakpoint hook but not be in a debugging session. Or only having a breakpoint in the second test and a timeout occurring in the first test.

[flub]

another reasonable way would be to sniff sys.gettrace() and sys.breakpointhook() to find out which debugger might be used. then try an api from the detected debugger to ask it whether it's currently in an active debugger session. This would again need more coordination with the debugger however. But I somehow suspect that any reliable solution is going to involve pytest(-timeout) and the debugger(s) to agree on some api they can use from each other. If we could do something like if sys.breakpointhook.__module__ == 'pydevd' and hasattr(sys.breakpointhook.__module__, 'is_debugging') and sys.breakpointhook.__module__.in_debugging_session() or something like that (less ugly of course) this would be better then trying to sniff without cooperation i think

[Mattwmaster58]

There is however a difference between a breakpoint hook being installed and actually being in an active debugging session isn't there? I think it's possible for someone having installed a breakpoint hook but not be in a debugging session.

Technically yes, but practically I can't think of that scenario occurring.

Or only having a breakpoint in the second test and a timeout occurring in the first test.

This is a side-effect that I'm willing to live with and had planned for. I think it's reasonable for a few reasons. One reason is that when you are debugging, chances are you are only looking a very small subset of tests, usually only one (based on my experience only). Another is that the alternative — allowing the one test to timeout — won't always continue to the second test (as noted in the readme):

Note that while by default ... py.test will continue to execute the tests after a test has timed, out this is not always possible

Finally, that last reason is my implementation would match the pytest_enter_pbd in terms of functionality ie indiscriminately disabling the timeout of all tests, regardless of where the breakpoint is set ie a breakpoint in test 1 will disable the timeout of test 2

Uggh, only read your first comment before writing all that.

WRT to using sys.gettrace(), here's a info from the documentation:

The gettrace() function is intended only for implementing debuggers, profilers, coverage tools and the like. Its behavior is part of the implementation platform, rather than part of the language definition, and thus may not be available in all Python implementations.

I tested measuring coverage as well as profiling a test test and found that no trace was set by either, but even so I've pushed a whitelist based approach of inspecting sys.gettrace. One side effect of this is that I think it will be a little trickier to make tests for this (I think).

[flub]

Thanks for your patience. I'm starting to like this implementation more, so that's good :)

I think for testing strategy it seems pydevd has a settrace function which can make it talk to a debugging server, whatever that is. So I suppose you could create a fixture which is a server, call settrace in the test, make the server continue the test, sleep for some seconds which definitely should time out then assert that the test finished successfully.

That's just a first brainstorm at how to go about this. I'm looking to test two things here: 1. The detection works and the timeout does get cancelled. 2. The whitelist keeps being correct.

[Mattwmaster58]

Wouldn't 2. The whitelist keeps being correct. be covered generally already by other tests?

[flub]

Wouldn't 2. The whitelist keeps being correct. be covered generally already by other tests?

Mostly yes, but I added that because I think it's important to test against the latest released version (probably in tox)

[sar772004]

Is ipdb.set_trace() also handled ? Since this also doesn't work in v1.3.4.

[Mattwmaster58]

@sar772004 Can you let me know if that works?

[Mattwmaster58]

I honestly have no idea why the tests are failing, and it's proving quite hard to debug😬 Classic. Don't know how to spell pdb.

[Mattwmaster58]

Alright, so here's the state of testing. It's easy enough to test pdb. We run into issues with pydevd and ipdb.

ipdb

Requires the testing test session disabled input capturing, but this isn't currently possible. I've submitted this PR to try and allow for this: pytest-dev/pytest#7207

pydevd

After jumping around the source code of pydevd.py I honestly have no clue what going on and consequently wasn't able to get the test for this working (even though it does it's job in PyCharm, ie, the timeout is cancelled). The test would break (as far as I could tell), but would still timeout with sys.get_trace.__module__ being None, for whatever reason. Ultimately I think that it wasn't connecting to the debug server correctly (or vice versa?) which caused the trace to not be set...which actually doesn't make sense because it did connect (I could tell cause it timed out). But I'm most likely wrong because I don't have almost any knowledge re PyDev's internal workings. (Do I use pydevd --client or pydevd --server? Who knows.)

I might take a look into ptvsd which might be a wrapper around PyDev.Debugger that may be able to tame it's power.

[flub]

You've typoed ipdb here. To be fair you changed the test to no longer skip if it's not there so may as well import directly, we do this for pexpect as well and it's not that a huge dependency.

[Mattwmaster58]

Yep, was focused on that one test. Will remove this.

[flub]

No longer used, which as I say above I think is fine.

[flub]

btw, after fixing the ipdb imports the ipdb test works just fine for me. It's only pydevd that'll need some more work.

[Mattwmaster58]

I must not have pushed my latest commits. If I actually check to make sure the test didn't error out ipdb will fail.

[Mattwmaster58]

I removed importing pexpect because pytest already checks internally to make sure pexpect has a spawn attribute before spawning child process.

[flub]

Hi, apologies for the break. Life can be hectic.

So this works, but has no working tests for ipdb or pydevd. ipdb would be fixed once we have a new pytest. Could we at least mark these tests as xfail? Currently the test suite doesn't pass at all. I'm a bit concerned that with an xfailing test no one will ever bother to work on fixing it though..

[Mattwmaster58]

I'm a bit concerned that with an xfailing test no one will ever bother to work on fixing it though.

Fair enough, I could definitely see that becoming the case.

[flub]

Could you rebase and fix the conflicts as well as xfail ipdb for now. There's no need to tie this to a pytest release itself for now and this is pretty much ready to be merged otherwise I think? Anything you'd still like to see addressed first?

[Mattwmaster58]

Nope, will rebase and xfail ipdb when I get the chance.

[Mattwmaster58]

(I'm showing my lack of git knowledge but) @flub I tried to rebase but there was quite a few merge conflicts, is there any reason I couldn't merge instead?

[flub]

Just had a look at doing this. Seems like there's already a merge
from master, so rebasing is not recommended. I suggested it because
it's my normal workflow but it doesn't matter that much. You can
merge master into this branch again and then we could squash merge
this whole thing into master so there's a single commit for the
feature if that's fine for you? I would like to have a somewhat
cleaner commit history then the current one... doesn't have to be
exactly one commit but squashing is by far the easiest by now.

I also don't mind doing the whole thing, but I'm not entirely sure on
how git will attribute all the commits and don't want to take away
your commits or re-write your history without you being fine with that.

[Mattwmaster58]

I don't care about the commit history that much. I'll remerge master.