TorchTests::test_delta_optimizer causes deadlocks in later tests

[maxhgerlach]

Environment:

1. Framework: PyTorch
2. Framework version: 1.10
3. Horovod version: master
4. MPI version: OpenMPI

Bug report:
With PyTorch 1.10 we occasionally observed deadlocks in TorchTests::test_dynamic_requires_grad. These went away by skipping test_delta_optimizer (the test before the previous test in alphabetical order), see #3291 (comment). Since that test only applies with MPI and when run on GPUs, only a few CI configurations can encounter the problem.

The skip is a workaround for some more fundamental problem that is not understood. It should be fixed and test_delta_optimizer should be re-eneabled then.

[Tixxx]

This is an interesting issue. I think this might be related to how the in-place tensor computations are done in the function hooks in delta optimizer. I can take a look if no on has started.