Work around TLS errors

[cognifloyd]

For some reason the ssl module is hitting an infinite recursion error. I thought this might be CircleCI-specific, but I was able to reproduce it locally with v3.6.13. The 3.7.0 version is supposed to be even more robust, so it could also be something about v3.6.13. I have not reproduced the error elsewhere, however. I was able to reproduce the error locally.

The error began appearing after we updated requests from v2.23.0 to v2.25.1 in StackStorm/st2#5215
In requests v2.24.0, it defaulted to using the python ssl module instead of PyOpenSSL due to various SELinux issues. Before that it used PyOpenSSL whenever it was installed. Since it worked with pyopenssl, we just revert to using it for the tests in this repo.
See: psf/requests#5443

[cognifloyd]

I was able to reproduce it locally. In order to fix this, I had to either downgrade to requests 2.23.0 or include the change from this PR.

[cognifloyd]

The recursion errors look like they could be caused by eventlet interacting poorly with the ssl lib.

I tried adding monkey_patch (as below) at the top of the test files without success.

from st2common.util.monkey_patch import monkey_patch
monkey_patch()

It looks like others have had issues on python3.6 + eventlet + ssl as well: eventlet/eventlet#371
In any case, forcing it to use the python implementaiton (which eventlet has presumably already patched) works.

[cognifloyd]

Hmm. If I add monkey_patch to tests/__init__.py that also resolves the issue. So something in nosetests must be importing requests or the ssl lib before monkey patching happens.

So, which fix is preferrable? monkey_patch in tests/__init__.py? or pyopenssl.inject_into_urllib3() to force re-importing ssl in urllib3 after the monkey_patching has already been done?

[blag]

Other than the generic reservations about monkeypatching anything to begin with (although I do understand that we have to here because we've painted ourselves into a bit of a corner with our use of eventlet), I don't see a problem with monkeypatching within tests/__init__.py, as long as the production code still actually works in production.

[cognifloyd]

K. I pushed a new commit that switches over to monkey patching in init.py. I agree that monkey patching is gross, but oh well.

[cognifloyd]

Well that didn't work on CircleCI. It might be working locally because of the st2 sources I used (that have some additional eventlet monkey patching added to the sources).

In any case, for production use, the monkey patching happens very early. With nosetest, we don't have quite as much control about when that monkey patching happens, so it's become a whak-a-mole game. I wonder if there's any way to get that monkey patching even earlier... hmm.

[cognifloyd]

Nope. I switched to a fresh st2 checkout. It still passes locally.

[cognifloyd]

I don't understand. Locally, moving the monkey_patch earlier resolved the issue. On CircleCI, even patching nosetests doesn't seem to resolve it. Any ideas?

[cognifloyd]

No matter where I put the monkey_patch, the infinite recursion error occurs on CircleCI. I was able to reproduce the infinite recursion error locally, but the monkey_patch fixed it for me.

So, the only thing that seems to fix the tests on circle CI is:

from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()

[cognifloyd]

@blag where do you want the hack for CircleCI? In the circle config like I did in the recent commits?

[blag]

Yeah, I think the CircleCI config is probably the best place for that.

[cognifloyd]

K. I cleaned up the debug commits. This should be ready to merge

[cognifloyd]

@blag can we merge this?

[blag]

LGTM