python_linux_dbg_native.py27.test.unit._reconnect_test.ReconnectTest failed

[jtattermusch]

https://grpc-testing.appspot.com/job/gRPC_pull_requests_linux/6437/testReport/junit/(root)/python_linux_dbg_native/py27_test_unit__reconnect_test_ReconnectTest/

++ dirname /var/local/git/grpc/tools/run_tests/helper_scripts/run_python.sh
+ cd /var/local/git/grpc/tools/run_tests/helper_scripts/../../..
++ realpath py27/bin/python
+ PYTHON=/var/local/git/grpc/py27/bin/python2.7
++ pwd
+ ROOT=/var/local/git/grpc
+ /var/local/git/grpc/py27/bin/python2.7 /var/local/git/grpc/src/python/grpcio_tests/setup.py test_lite
running test_lite
Running       unit._reconnect_test.ReconnectTest.test_reconnect
[95mTesting gRPC Python...
[0m[91mERROR         unit._reconnect_test.ReconnectTest.test_reconnect
[0m1 tests finished:
	0 successful
	1 unsuccessful
	0 skipped
	0 expected failures
	0 unexpected successes
Interrupted Tests:
	[]

Errors/Failures: 
[91munit._reconnect_test.ReconnectTest.test_reconnect[0m
[1mtraceback:[0m
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/var/local/git/grpc/src/python/grpcio_tests/tests/unit/_reconnect_test.py", line 51, in test_reconnect
    self.assertEqual(_RESPONSE, multi_callable(_REQUEST))
  File "/var/local/git/grpc/py27/local/lib/python2.7/site-packages/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/var/local/git/grpc/py27/local/lib/python2.7/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
_Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Endpoint read failed)>

[jtattermusch]

still seeing these on master https://sponge.corp.google.com/target?id=3086cce2-aadb-4f53-b2db-91dcbd02eac2&target=github/grpc/python_linux_opt_native&searchFor=&show=FAILED&sortBy=STATUS

[jtattermusch]

++ dirname /var/local/git/grpc/tools/run_tests/helper_scripts/run_python.sh
+ cd /var/local/git/grpc/tools/run_tests/helper_scripts/../../..
++ realpath py27/bin/python
+ PYTHON=/var/local/git/grpc/py27/bin/python2.7
++ pwd
+ ROOT=/var/local/git/grpc
+ /var/local/git/grpc/py27/bin/python2.7 /var/local/git/grpc/src/python/grpcio_tests/setup.py test_lite
running test_lite
Running       unit._reconnect_test.ReconnectTest.test_reconnect
[95mTesting gRPC Python...
[0m[91mERROR         unit._reconnect_test.ReconnectTest.test_reconnect
[0m1 tests finished:
	0 successful
	1 unsuccessful
	0 skipped
	0 expected failures
	0 unexpected successes
Interrupted Tests:
	[]

Errors/Failures: 
[91munit._reconnect_test.ReconnectTest.test_reconnect[0m
[1mtraceback:[0m
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/var/local/git/grpc/src/python/grpcio_tests/tests/unit/_reconnect_test.py", line 51, in test_reconnect
    self.assertEqual(_RESPONSE, multi_callable(_REQUEST))
  File "/var/local/git/grpc/py27/local/lib/python2.7/site-packages/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/var/local/git/grpc/py27/local/lib/python2.7/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
_Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Endpoint read failed)>

[1mstdout:[0m

[1mstderr:[0m

[jtattermusch]

This is still happening, currently it's failing 6% of all multilang tests (= our top flake for wrapped languages).

• only happening for py27
• happening both for dbg and opt, but more common for opt

Recent failures:
opt:
https://sponge.corp.google.com/invocation?id=428b248c-c976-4182-87d2-4325b0b5f406
https://sponge.corp.google.com/invocation?id=c9cbadbd-f834-4403-bef7-5b8b4b56d538
https://sponge.corp.google.com/invocation?id=3e95f23b-e4b2-4a93-985c-c8fea34dcf98
https://sponge.corp.google.com/invocation?id=48b1cd59-9139-499d-badb-32afe2171e4a

dbg:
https://sponge.corp.google.com/invocation?id=034713c8-f554-4037-9611-8c1395e9a56d
https://sponge.corp.google.com/invocation?id=53f70563-5020-44bf-aa13-346fd1dd531a
https://sponge.corp.google.com/invocation?id=bf6516fe-66f6-4659-80b5-d60ab5b9cf6b

[jtattermusch]

@kpayson64, can you take a look?

[jtattermusch]

@kpayson64, any progress here?

[jtattermusch]

@mehrdada , could you take a look?

[kpayson64]

@jtattermusch
Sorry, I've been looking at #13327 last week. Maybe @mehrdada can take a look.

[mehrdada]

Sure, will take a look

[mehrdada]

For future reference, to summarize the issue, some of those invocations are general python infra failures, and at least one of the Sponge logs affects Python3.4, so it is not just 2.7. Looks like the UNAVAILABLE errors that we used to see that went away by disabling so_reuseport:

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/var/local/git/grpc/src/python/grpcio_tests/tests/unit/_reconnect_test.py", line 94, in test_reconnect
    self.assertEqual(_RESPONSE, multi_callable(_REQUEST))
  File "/var/local/git/grpc/py27/local/lib/python2.7/site-packages/grpc/_channel.py", line 487, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/var/local/git/grpc/py27/local/lib/python2.7/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
_Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Socket closed)>