Random test failures on some tests on master

[reversefold]

I've been working on a patch and was trying to get a baseline for the test suites using tox on the master branch. I'm getting random hangs and test failures with the dummyserver tests. They will sometimes pass and sometimes fail. Here's an example of a failure:

________________________________________________________________________________________________ TestHTTPProxyManager.test_proxy_conn_fail _________________________________________________________________________________________________

self = <test.with_dummyserver.test_proxy_poolmanager.TestHTTPProxyManager testMethod=test_proxy_conn_fail>

    def test_proxy_conn_fail(self):
        host, port = get_unreachable_address()
        http = proxy_from_url('http://%s:%s/' % (host, port), retries=1, timeout=0.05)
        self.addCleanup(http.clear)                                                                                                                                                                                                                 self.assertRaises(MaxRetryError, http.request, 'GET',
                          '%s/' % self.https_url)
        self.assertRaises(MaxRetryError, http.request, 'GET',
                          '%s/' % self.http_url)

        try:
            http.request('GET', '%s/' % self.http_url)
            self.fail("Failed to raise retry error.")
        except MaxRetryError as e:
>           self.assertEqual(type(e.reason), ProxyError)
E           AssertionError: <class 'urllib3.exceptions.ConnectTimeoutError'> != <class 'urllib3.exceptions.ProxyError'>
                                                                                                                                                                                                                                            test/with_dummyserver/test_proxy_poolmanager.py:66: AssertionError                                                                                                                                                                          ----------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------
Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x111c7d610>, 'Connection to hmxysfyfmsecholvhzjufgbxfncwr
rtprlqjmcxktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': /
Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x111c7d610>, 'Connection to hmxysfyfmsecholvhzjufgbxfncwr
rtprlqjmcxktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': /
Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x111c7de90>, 'Connection to hmxysfyfmsecholvhzjufgbxfncwrrtprlqjmc
xktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': http://localhost:58795/
Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x111c7de90>, 'Connection to hmxysfyfmsecholvhzjufgbxfncwrrtprlqjmc
xktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': http://localhost:58795/
Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x111c7ded0>, 'Connection to hmxysfyfmsecholvhzjufgbxfncwrrtprlqjmc
xktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': http://localhost:58795/
Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x111c7ded0>, 'Connection to hmxysfyfmsecholvhzjufgbxfncwrrtprlqjmc
xktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': http://localhost:58795/
----------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x111c7d610>, 'Connection t
o hmxysfyfmsecholvhzjufgbxfncwrrtprlqjmcxktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x111c7de90>, 'Connection to hmxysfy
fmsecholvhzjufgbxfncwrrtprlqjmcxktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': http://localhost:58795/
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x111c7ded0>, 'Connection to hmxysfy
fmsecholvhzjufgbxfncwrrtprlqjmcxktvhjdwzuydlfqyqaaplb timed out. (connect timeout=0.05)')': http://localhost:58795/
============================================================================================================= warnings summary =============================================================================================================
test/test_util.py::TestUtil::()::test_ssl_wrap_socket_loads_the_cert_chain
  /Users/papercrane/src/urllib3/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present
an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    SNIMissingWarning

test/contrib/test_pyopenssl.py::TestHTTPS::test_client_intermediate
  /Users/papercrane/src/urllib3/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-u
sage.html#ssl-warnings
    InsecureRequestWarning)
                                                                                                                                                                                                                                            test/contrib/test_socks.py::TestSocks5Proxy::test_basic_request                                                                                                                                                                               /Users/papercrane/src/urllib3/dummyserver/server.py:131: NoIPv6Warning: No IPv6 support. Falling back to IPv4.
    NoIPv6Warning)

test/contrib/test_socks.py::TestSOCKSWithTLS::test_basic_request
  /Users/papercrane/src/urllib3/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You ca
n upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    InsecurePlatformWarning

test/with_dummyserver/test_connectionpool.py::TestRetryPoolSize::test_pool_size_retry
  /Users/papercrane/src/urllib3/test/with_dummyserver/test_connectionpool.py:1003: PendingDeprecationWarning: Please use assertEqual instead.
    self.assertEquals(self.pool.num_connections, 1)

test/with_dummyserver/test_https.py::TestHTTPS_IPv6Addr::test_strip_square_brackets_before_validating
  /Users/papercrane/src/urllib3/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for ::1 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and depreca
ted by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
    SubjectAltNameWarning

test/with_dummyserver/test_socketlevel.py::TestRetryPoolSizeDrainFail::test_pool_size_retry_drain_fail
  /Users/papercrane/src/urllib3/test/with_dummyserver/test_socketlevel.py:1471: PendingDeprecationWarning: Please use assertEqual instead.
    self.assertEquals(pool.num_connections, 1)

-- Docs: http://doc.pytest.org/en/latest/warnings.html

I'm also seeing random failures with test/contrib/test_securetransport.py and some of the other ssl based tests.

[Lukasa]

Yup, sadly this is a bit of a known problem. It's particularly common with slower test environments because several of the tests run with short timeouts or are expected to run in a fairly straightforward environment.

These are tricky to fix. Because they're timing related, the only way to get them to behave the way we actually want in a reliable manner is to write a complete harness for the test environment that mocks out almost any I/O or timing syscalls. This is quite the endeavour, so we normally just tolerate some minor flakiness, especially as it tends only to be exposed in limited environments.

[reversefold]

Ok, thanks for the explanation. I'm surprised that I'd be running into these failures, though. I'm running on a MacBookPro 2.8Ghz i7. I'm also doing some other things on the machine, but it's usually fine for development.

I've also been testing in a xenial vagrant box and am seeing similar issues.

One last thing I've noticed is that the pypy tests seem to crash during the ssl tests. The last line of output looks like this:

test/contrib/test_pyopenssl.py ................ERROR: InvocationError: '/Users/papercrane/src/urllib3/.tox/pypy/bin/py.test --cov urllib3 test'

I've tried excluding that test and the next set of ssl tests exhibit the same behavior.

[Lukasa]

Interesting. I haven't been able to reproduce this at all. What PyPy are you using?

[reversefold]

pypy2-5.7.1 installed by pyenv. I also had to install cryptography specially to have it able to include opensslv.h from macports:

./.tox/pypy/bin/pip install 'cryptography>=1.3.4' --global-option=build_ext --global-option="-L/opt/local/lib" --global-option="-I/opt/local/include"

[reversefold]

I switched to pypy-5.4.1, installed by pyenv, and the crash doesn't happen, so it seems it's an issue with the newer pypy.

[pquentin]

This issue does not appear to be actionable, and I believe it can be closed in favor of #1595 and #1706.