Difficulty debugging testtools.TestCase tests

[wshanks]

Consider this test file:

import unittest


class TestStuff(unittest.TestCase):
    def test_stuff(self):
        a = 0
        self.assertEqual(a, 1)

Running this with pytest --pdb test_stuff.py and then entering where at the pdb prompt gives:

test_stuff.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <test_stuff.TestStuff testMethod=test_stuff>

    def test_stuff(self):
        a = 0
>       self.assertEqual(a, 1)
E       AssertionError: 0 != 1

test_stuff.py:7: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /lib/python3.11/unittest/case.py(866)_baseAssertEqual()
-> raise self.failureException(msg)
(Pdb) where
  /lib/python3.11/unittest/case.py(57)testPartExecutor()
-> yield
  /lib/python3.11/unittest/case.py(623)run()
-> self._callTestMethod(testMethod)
  /lib/python3.11/unittest/case.py(579)_callTestMethod()
-> if method() is not None:
  /tmp/test_stuff.py(7)test_stuff()
-> self.assertEqual(a, 1)
  /lib/python3.11/unittest/case.py(873)assertEqual()
-> assertion_func(first, second, msg=msg)
> /lib/python3.11/unittest/case.py(866)_baseAssertEqual()
-> raise self.failureException(msg)
(Pdb) 

Now consider this alternate test module with unittest replaced with testtools:

import testtools


class TestStuff(testtools.TestCase):
    def test_stuff(self):
        a = 0
        self.assertEqual(a, 1)

Running with pytest --pdb test_stuff.py gives now:

test_stuff.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
'NoneType' object is not iterable

During handling of the above exception, another exception occurred:
NOTE: Incompatible Exception Representation, displaying natively:

testtools.testresult.real._StringException: traceback-1: {{{
ValueError: In File: /tmp/test_stuff.py
TestCase.tearDown was not called. Have you upcalled all the way up the hierarchy from your tearDown? e.g. Call super(TestStuff, self).tearDown() from your tearDown().
}}}

Traceback (most recent call last):
  File "/tmp/test_stuff.py", line 9, in test_stuff
    self.assertEqual(a, 1)
  File "/lib/python3.11/site-packages/testtools/testcase.py", line 394, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/lib/python3.11/site-packages/testtools/testcase.py", line 481, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 0 != 1
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /lib/python3.11/site-packages/_pytest/unittest.py(234)_addexcinfo()
-> fail("".join(values), pytrace=False)
(Pdb) where
> /lib/python3.11/site-packages/_pytest/unittest.py(234)_addexcinfo()
-> fail("".join(values), pytrace=False)
  /lib/python3.11/site-packages/_pytest/outcomes.py(198)fail()
-> raise Failed(msg=reason, pytrace=pytrace)
(Pdb) 

In the unittest case, the debugger is opened in the context of the assertion error that is raised. In the testtools case, the debugger is opened in a pytest context and the stack does contain the code from the test module. Also, there is a message about some other exception (NoneType object is not iterable). Is this a bug, or just an incompatibility between pytest and testtools? Is there any way to get the debugger to open on the assertion error with testtools like it does with unittest?

[RonnyPfannschmidt]

pytest is entirely unaware of the testtools extensions, there are no plans to support them as of now

[wshanks]

Thanks! With that hint, I looked at the pytest and testtools code a little more closely to understand what was happening and see if anything could be done to fix things. I was hoping there might be a solution like a small tweak to testtools or a pytest extension. Here is what I found in case it is helpful to someone else:

1. pytest has special code for handling unittest (as alluded to by the comment about not being aware of testtools) that overrides many of the TestCase methods (here). Since testtools.TestCase subclasses unittest.TestCase, this code mostly works for it as well but not entirely.
2. testtools has exception handling code (here) that passes the exception info to self.onException and saves only the Exception part (item 1 of the tuple).
3. self.onException calls self.addDetail with the traceback part of the exception info passed into a TracebackContent class here. That formats the traceback as a string-like object and drops the original traceback object.
4. testtools then tries to call self.addFailure passing the exception info it gets from self._details_to_exc_info here. self._details_to_exc_info gives back an exception info tuple with a string exception class instance containing the traceback in its repr as item 1 and None for item 2 where the traceback would be.
5. addFailure calls self._addexcinfo() which is one of the methods pytest overrides. It explicitly tries to access the traceback from the exception info here which then hits the "NoneType not iterable error" here trying to pass the None entry from the exception info to Traceback.

I don't think there is an easy solution. I tried hacking in something to save the traceback object and pass it when pytest needs it, but that didn't open the debugger in the right location. I think that was because testtools does extra processing and by the time the debugger is activated the stack is gone. Probably more methods of testtools could be patched like they are for unittest but that seems painful to maintain.

In practical terms, in some cases, the base test class can be swapped back from testtools.TestCase to unittest.TestCase without too much editing. (Or you can migrate to pytest 🙂 but I was mainly interested in using pytest tools to debug tests in a project that I don't own).