Getting 'Error: 13 INTERNAL: No message received' when sending/receiving JSON message that are 60 - 80 kb #2707
Comments
|
I published a change in version 1.10.6 that I think will fix this. Please try it out. |
|
@murgatroid99 Thank you for a quick response. Unfortunately the change did not seem to work. We are still receiving the OK status on the gRPC client side before we receive the data. |
|
Can you share the code you are using to reproduce this error? |
|
Here's the client/server code |
|
sorry I have updated the file above @murgatroid99 |
|
That code is configured to communicate on localhost. Do you ever actually reproduce this error on localhost? |
|
when both client/server are hosted locally, nope. Only when server is hosted on a remote server |
|
OK, while I'm figuring out how to reproduce that myself, can you run you test server with the environment variables |
|
D 2024-04-03T18:33:15.742Z | v1.10.6 296 | server | (1) Attempting to bind 0.0.0.0:4006 |
|
Here is a comparison of the logs when running it successfully locally. Main difference is this line here - D 2024-04-03T18:54:46.687Z | v1.10.6 39072 | server | (1) Server constructed |
|
That's not the difference. The first log has the line |
|
I suspect that this is a bug in Node. The gRPC code in question waits for the |
|
Ok this wasn't occurring with the deprecated grpc pkg. Also, we noticed that grpc-js sends response in multiple chunks over TCP. The legacy grpc sends the response in a single chunk over RSL protocol. You think this possibly be related to the issue? |
|
Btw were you able to reproduce? Also, do you think the 150ms timeout is a viable solution? |
That package didn't use the Node networking stack. If this is a bug in Node, that difference would be expected.
This is how HTTP/2 works. Data frames have a maximum size, and gRPC messages that are larger than that size are divided into multiple data frames.
I have no idea what this is referring to. Both libraries use the same protocol (gRPC over HTTP/2), which includes the same chunking rules. I have never even heard of the RSL protocol.
No. All of the operations involved in a gRPC stream are deterministic and sequential (to the extent that we care about), so it should be possible to get it right consistently without fudging the timing. There is no reason to believe that any particular timeout will consistently prevent this error, and 150ms will substantially harm performance in every other case. |
I don't yet have the setup to reproduce this. However, I did realize that we have an existing test that involves sending a large payload (~300KB), so I'd like to understand why that didn't already catch this error
|
|
Ok so we tested it on TLS and it works! Also, it works on Linux with non-TLS. Maybe adding tests for non-TLS will catch this error in the future. Thanks for all of the help! |
Problem description
When sending JSON string that are 60 - 80 kb long, we are seeing intermittent 'Error: 13 INTERNAL: No message received' on the gRPC client server. Based on our investigation it is happening due to gRPC client receiving the Ok status call before it receives the message. When we added in the below setTimeout call the issue stopped happening.
Below is the exact error message
Error: Error: 13 INTERNAL: No message received at callErrorFromStatus (C:\grpc-test\node_modules\@grpc\grpc-js\build\src\call.js:31:19) at Object.onReceiveStatus (C:\grpc-test\node_modules\@grpc\grpc-js\build\src\client.js:180:80) at Object.onReceiveStatus (C:\grpc-test\node_modules\@grpc\grpc-js\build\src\client-interceptors.js:360:141) at Object.onReceiveStatus (C:\grpc-test\node_modules\@grpc\grpc-js\build\src\client-interceptors.js:323:181) at C:\grpc-test\node_modules\@grpc\grpc-js\build\src\resolving-call.js:99:78 at process.processTicksAndRejections (node:internal/process/task_queues:77:11) for call at at ServiceClientImpl.makeUnaryRequest (C:\grpc-test\node_modules\@grpc\grpc-js\build\src\client.js:160:32) at ServiceClientImpl.testMethod (C:\grpc-test\node_modules\@grpc\grpc-js\build\src\make-client.js:105:19) at C:\grpc-test\dist\client.js:25:16 at step (C:\grpc-test\node_modules\tslib\tslib.js:195:27) at Object.next (C:\grpc-test\node_modules\tslib\tslib.js:176:57) at C:\grpc-test\node_modules\tslib\tslib.js:169:75 at new Promise (<anonymous>) at Object.__awaiter (C:\grpc-test\node_modules\tslib\tslib.js:165:16) at C:\grpc-test\dist\client.js:22:60 at Layer.handle [as handle_request] (C:\grpc-test\node_modules\express\lib\router\layer.js:95:5) { code: 13, details: 'No message received', metadata: Metadata { internalRepr: Map(0) {}, options: {} } }Reproduction steps
We setup gRPC client and gRPC server on two different servers and sent JSON string with 7k line count over the network. When we did this the above issue was reproducible.
Environment
Additional context
The issue started happening when we migrated over to grpc-js from deprecated grpc package.
The text was updated successfully, but these errors were encountered: