Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

grpc: received message larger than max #203

Open
kakkoyun opened this issue Jan 10, 2022 · 3 comments · May be fixed by #836
Open

grpc: received message larger than max #203

kakkoyun opened this issue Jan 10, 2022 · 3 comments · May be fixed by #836
Labels
area/rpc Something about RPC calls to the server-side bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed

Comments

@kakkoyun
Copy link
Member

Opening this in here but this could be solved on the agent side as well. This is probably an overlooked side effect of Write Request batching on the agent side. #116 We probably need to communicate the max message size to the agent (with a flag or an exposed discovery API?)

I see another message now, a lot of them.

level=error ts=2022-01-10T13:20:04.693514836Z caller=write_client.go:83 msg="Writeclient failed to send profiles" err="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (174243321 vs. 4194304)"

Originally posted by @korjavin in parca-dev/parca#514 (comment)

cc @Sylfrena @brancz
@brancz
Copy link
Member

brancz commented Jan 11, 2022

We're using the default size so I think we don't need to make this configurable yet. I believe we can check the size of the proto message before and after appending and if doubling the message that was just appended would overflow the size we should just send it out instead of waiting for the interval to kick in.

@kakkoyun
Copy link
Member Author

Sounds like a good strategy. I'm moving the issue to agent side.

@kakkoyun kakkoyun transferred this issue from parca-dev/parca Jan 11, 2022
@kakkoyun kakkoyun added the bug Something isn't working label Jan 13, 2022
@kakkoyun kakkoyun assigned kakkoyun and Sylfrena and unassigned kakkoyun Mar 17, 2022
@kakkoyun kakkoyun added P0 The most important and urgent and removed P0 The most important and urgent labels Mar 18, 2022
@kakkoyun kakkoyun mentioned this issue May 30, 2022
Closed
@brancz
Copy link
Member

brancz commented May 31, 2022

Ultimately though, I think we should stream the data, but that we should only do once we truly implement our own protocol/format.

@kakkoyun kakkoyun added the area/rpc Something about RPC calls to the server-side label Jun 20, 2022
@kakkoyun kakkoyun added this to the v0.10.0 milestone Aug 2, 2022
@kakkoyun kakkoyun added good first issue Good for newcomers help wanted Extra attention is needed labels Aug 3, 2022
heylongdacoder added a commit to heylongdacoder/parca-agent that referenced this issue Sep 24, 2022
@heylongdacoder
pkg/agent: trigger batch before grpc message size overflow
e2ce6ce 
Fixes: parca-dev#203

Signed-off-by: heylongdacoder <heylongdacoder@gmail.com>
@heylongdacoder heylongdacoder linked a pull request Sep 24, 2022 that will close this issue
Open
@Sylfrena Sylfrena removed their assignment Oct 18, 2022
@kakkoyun kakkoyun modified the milestones: v0.10.0, v0.11.0 Nov 2, 2022
@kakkoyun kakkoyun removed this from the v0.12.0 milestone Mar 15, 2023
@kakkoyun kakkoyun self-assigned this May 17, 2023
@kakkoyun kakkoyun removed their assignment Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/rpc Something about RPC calls to the server-side bug Something isn't working good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pkg/agent: trigger batch before grpc message size overflow heylongdacoder/parca-agent
3 participants
@kakkoyun @brancz @Sylfrena