Exception not retrieved: grpc._cython.cygrpc.ExecuteBatchError: Failed "execute_batch

[AMontagu]

What version of gRPC and what language are you using?

Python 3.8
grpcio 1.48.1

What operating system (Linux, Windows,...) and version?

Linux in docker. Debian slim buster

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.8

What did you do?

Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.

I have a UnaryStream and inside this UnaryStream I call an other UnaryStream from an other service.

package app1.report;

service ReportController {
    rpc Generate(ReportGenerateRequestRequest) returns (stream ReportGenerateResponseResponse) {}
}

package app2.report;

service ReportController {
    rpc Create(ReportCreateMessageRequest) returns (stream ReportCreateStreamResponse) {}
}

It's basically just api composition to retrieve the correct datas before creating the report.

In the app1 Generate method I have something like:

data, metadata = report_builder.build_report_data()
# Call report generator.
async for response in ReportGeneratorGrpcApi().create(data, metadata):
  # SOME BUSINESS LOGIC HERE
  await context.write(
    ParseDict(
        {
            "message": response.message,
            "code": response.code,
            "report": {
                "status": response.data.status,
                "uuid": response.data.uuid,
            },
        },
        report_pb2.ReportGenerateResponseResponse(),
    )
  )

What did you expect to see?

After reading #25339 I am thinking that I have a non handled exception in my business logic. So I expect to see the correct exception to handle it.

Will try tomorrow to raise, on purpose, an exception in my local environment to test if the guess is correct.

What did you see instead?

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django_socio_grpc/servicer_proxy.py", line 50, in async_handler
    return await instance_action(
  File "/opt/code/report/services/report_service.py", line 168, in Generate
    return await context.write(
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 142, in write
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/callback_common.pyx.pxi", line 147, in _send_message
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/callback_common.pyx.pxi", line 98, in execute_batch
grpc._cython.cygrpc.ExecuteBatchError: Failed "execute_batch": (<grpc._cython.cygrpc.SendMessageOperation object at 0x7fbb74965c30>,)

Anything else we should know about your project / environment?

[AMontagu]

Hello
I test tested to raise an Exception in my business logic:

data, metadata = report_builder.build_report_data()
# Call report generator.
async for response in ReportGeneratorGrpcApi().create(data, metadata):
  # SOME BUSINESS LOGIC HERE
  raise Exception("test")
  await context.write(
    ParseDict(
        {
            "message": response.message,
            "code": response.code,
            "report": {
                "status": response.data.status,
                "uuid": response.data.uuid,
            },
        },
        report_pb2.ReportGenerateResponseResponse(),
    )
  )

But the error is correctly handled. So it should come from an other issue. Following the traceback I discovered that I was wrong about the place of the exception. The error appear on the context.write in my exception handling.
I adeed more debug to understand what kind of exception it is and on the state of the context.

try:
  data, metadata = report_builder.build_report_data()
  # Call report generator.
  async for response in ReportGeneratorGrpcApi().create(data, metadata):
    # SOME BUSINESS LOGIC HERE
    raise Exception("test")
    await context.write(
      ParseDict(
          {
              "message": response.message,
              "code": response.code,
              "report": {
                  "status": response.data.status,
                  "uuid": response.data.uuid,
              },
          },
          report_pb2.ReportGenerateResponseResponse(),
      )
    )
except Exception:
  # Exception append on next line
  return await context.write(
      ParseDict(
          {
              "message": "Unknow error on report generation",
              "code": ReportMessageCode.ERROR,
              "report": {"status": ReportStatuses.ERROR},
          },
          report_pb2.ReportGenerateResponseResponse(),
      )
  )

[XuanWang-Amos]

It's similar to this issue: #31570.

We'll need more logs to debug core failure, please comment here with logs if anyone is able to reproduce with additional environment flags:
GRPC_VERBOSITY=debug GRPC_TRACE=all,-timer,-timer_check

[AMontagu]

Hello @XuanWang-Amos

We have a big roadmap until the end of january. Not possible for us to provide more log now. Will try as soon as teams stop working on the project that produce this error.

[njhill]

I've reproduced with debug logs: #31570 (comment)

[XuanWang-Amos]

Closing since this should be fixed with this PR: #32551.