New error after migrating to @grpc/grpc-js -> RESOURCE_EXHAUSTED: Bandwidth exhausted

[sjkummer]

Problem description

As proposed, we have migrated to the new @grpc/grpc-js npm package. Now, we got a new RESOURCE_EXHAUSTED error every now and then on the client. Unfortunately, nothing is logged on the server side (which was migrated to @grpc/grpc-js too). Also, there is very little documentation available, e.g.: are there potential grpc settings which may limit something? How to get more logs?

Reproduction steps

Unclear. Occurs about once a day. Did not occur before migration.

Environment

• Server OS name, version and architecture: Linux Ubuntu 18.04 amd64
• Client OS name, version and architecture: Alpine arm64 (Dockerized)
• Node version: 12
• Package name and version: @grpc/grpc-js 1.3.7

Additional context

Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
    at Object.callErrorFromStatus (/app/backend/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/app/backend/node_modules/@grpc/grpc-js/build/src/client.js:179:52)
    at Object.onReceiveStatus (/app/backend/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/app/backend/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /app/backend/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (internal/process/task_queues.js:79:11)

[murgatroid99]

I suggest setting the channel option grpc-node.max_session_memory. It was added in #1666 because other users reported that it could resolve RESOURCE_EXHAUSTED errors. The value is in number of megabytes, and the default is 10. I suggest setting it to a large value that does not exceed the memory available to you, maybe 500 or 1000.

[sjkummer]

Thanks, I'll try that.

Two questions:

• Isn't Bandwidth exhausted a missleading error message, if this is related to memory? Should I file a separate bug for that?
• 10 MB seems like a very low default value to me ... should't this be in relation to the configured node memory limt?

[murgatroid99]

Unfortunately, the information that Node gives gRPC for these errors is indistinguishable from an actual resource exhausted error. I could broaden the message a little to include this kind of failure.

The 10 MB default is also set by Node itself. I think it's not really obvious what the right value is because it's a per-session value, so the total memory usage limit depends on how many different sessions you intend to have. gRPC abstracts that away somewhat, but it's roughly the number of different servers you are connecting to.

[sjkummer]

Thanks for the clarifications. I'll report back if I see the error again with the next two weeks after having set a hight limit. Otherwise, the ticket can be closed.