test_signal_handlers_restored_in_teardown failing on mac and linux

[akihironitta]

🐛 Bug

In all ongoing PRs,test_signal_handlers_restored_in_teardown fails, and a few test cases after, the job is terminated.

tests/trainer/connectors/test_signal_connector.py::test_signal_handlers_restored_in_teardown FAILED [ 79%]
tests/trainer/connectors/test_signal_connector.py::test_fault_tolerant_sig_handler[False-False] PASSED [ 79%]
tests/trainer/connectors/test_signal_connector.py::test_fault_tolerant_sig_handler[False-True] PASSED [ 79%]
Terminated
tests/trainer/connectors/test_signal_connector.py::test_fault_tolerant_sig_handler[True-False] 
Error: Process completed with exit code 143.

To Reproduce

https://github.com/PyTorchLightning/pytorch-lightning/actions/runs/1751182495

Expected behavior

• No test failure of the test case above.
• No termination of CI jobs

Additional context

cc @tchaton @rohitgr7 @akihironitta @carmocca @Borda

[akihironitta]

Here's the minimal reproducible command and its error msg:

$ coverage --version
Coverage.py, version 6.3 with C extension
Full documentation is at https://coverage.readthedocs.io
$ coverage run -m pytest tests/trainer/connectors/test_signal_connector.py::test_signal_handlers_restored_in_teardown -vv
=============================================================== test session starts ===============================================================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/nitta/.pyenv/versions/miniconda3-latest/envs/dev-pl39/bin/python
cachedir: .pytest_cache
rootdir: /home/nitta/work/github.com/PyTorchLightning/pytorch-lightning, configfile: setup.cfg
plugins: rerunfailures-10.2, hydra-core-1.1.1
collected 1 item                                                                                                                                  

tests/trainer/connectors/test_signal_connector.py::test_signal_handlers_restored_in_teardown FAILED                                         [100%]

==================================================================== FAILURES =====================================================================
____________________________________________________ test_signal_handlers_restored_in_teardown ____________________________________________________

    @RunIf(skip_windows=True)
    @mock.patch.dict(os.environ, {"PL_FAULT_TOLERANT_TRAINING": "1"})
    def test_signal_handlers_restored_in_teardown():
        """Test that the SignalConnector restores the previously configured handler on teardown."""
>       assert signal.getsignal(signal.SIGTERM) is signal.SIG_DFL
E       assert <bound method Coverage._on_sigterm of <coverage.control.Coverage object at 0x7f75a9ee05b0>> is <Handlers.SIG_DFL: 0>
E        +  where <bound method Coverage._on_sigterm of <coverage.control.Coverage object at 0x7f75a9ee05b0>> = <function getsignal at 0x7f75a9d98820>(<Signals.SIGTERM: 15>)
E        +    where <function getsignal at 0x7f75a9d98820> = signal.getsignal
E        +    and   <Signals.SIGTERM: 15> = signal.SIGTERM
E        +  and   <Handlers.SIG_DFL: 0> = signal.SIG_DFL

tests/trainer/connectors/test_signal_connector.py:34: AssertionError
============================================================= short test summary info =============================================================
FAILED tests/trainer/connectors/test_signal_connector.py::test_signal_handlers_restored_in_teardown - assert <bound method Coverage._on_sigterm ...
========================================================== 1 failed, 4 warnings in 0.53s ==========================================================

[akihironitta]

This error comes from coverage==6.3 and could probably be related to

• Process hang with Coverage 6.3 nedbat/coveragepy#1310
• coverage 6.3 failed for parallel execution  nedbat/coveragepy#1312

I'll look into it, but for now, let's use the previous version coverage<6.3.