[Python] Handle BlockingIOError if multiple loops are bound to PollerCompletionQueue

[maffoo]

Fixes #25364

When using grpc with multiple asyncio event loops in different threads, we may have multiple loops bound to the same PollerCompletionQueue. If the event loops support file descriptor monitoring, each loop will be configured with loop.add_reader to handle read events on the notification socket. This means that when completion events are available, PollerCompletionQueue._handle_events may be called multiple times for the different loops, but only one call will actually receive the byte written to the notification socket. The other calls will raise BlockingIOError, which then bubbles up and gets printed by the event loop's default exception handler, resulting in spurious error messages since things are in fact working fine. This changes _handle_events to catch and supress these BlockingIOErrors.

[gnossen]

Thanks for the contribution! I don't believe we currently have any tests that cover operating with multiple event loops. Do you think you could help by contribution one that replicates your setup?

[maffoo]

@gnossen, I added a basic test of using grpc.aio from multiple threads, please take a look. I haven't been able to run this locally because grpc is failing to build for me. I tried to follow the instructions and ran

python tools/run_tests/run_tests.py -l python --compiler python3.12

in a python 3.12 virtualenv, but that fails with an error while building wheels. Any additional pointers you have about running tests locally would be greatly appreciated.

[XuanWang-Amos]

Hi, thanks for contributing the test! Do you mind share more information about the failures? In the meanwhile, you can also trigger the test using bazel like this:

bazel test "//src/python/grpcio_tests/tests_aio/unit:multithread_test"

[XuanWang-Amos]

Looks like the new test is timing out, do you mind take a look?

[gnossen]

Looks like the distribtest failure is an unrelated non-hermetic regression on the master branch unrelated to this PR. (cause). We'll take care of that out of band.