Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ErrMessageSizeTooLarge indicates a kafka server response when the error is in the Producer #2137

Closed
shermanCRL opened this issue Feb 9, 2022 · 2 comments · Fixed by #2628 · May be fixed by #2851
Closed

ErrMessageSizeTooLarge indicates a kafka server response when the error is in the Producer #2137

shermanCRL opened this issue Feb 9, 2022 · 2 comments · Fixed by #2628 · May be fixed by #2851
Labels
bug needs-investigation Issues that require followup from maintainers

Comments

@shermanCRL
Copy link

shermanCRL commented Feb 9, 2022 •
edited by dnwe

Versions
Sarama Kafka Go
1.29.0 any 1.17
Configuration

What configuration values are you using for Sarama and Kafka?

defaults
Logs

Logs aren’t especially important here (not a bug), except to say that the relevant error is kafka server: Message was too large, server rejected it to avoid allocation error.

Problem Description

When the message size exceeds Producer.MaxMessageBytes, the error message indicates that the error is coming from the Kafka server, when in fact it’s a Producer error.

In our application, we recently spent a fair amount of time working with a customer, chasing down this issue -- we unproductively focused on the server while the error was in our software. cockroachdb/cockroach#76258. The downstream server had a very large max message size (hundreds of MB), but the errors persisted, and it took a while to recognize that Producer.MaxMessageBytes (defaulted to 1MB) was the culprit.

We’d like to see a more accurate and descriptive error message.

Thanks for the consideration, I can mock up a PR to illustrate a preferred solution if it would be helpful.
@dnwe dnwe added the bug label May 11, 2022
@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions github-actions bot added the stale Issues and pull requests without any recent activity label Aug 18, 2023
@dnwe dnwe added needs-investigation Issues that require followup from maintainers and removed stale Issues and pull requests without any recent activity labels Aug 18, 2023
@dnwe dnwe mentioned this issue Aug 29, 2023
Merged
@dnwe
Copy link
Collaborator

dnwe commented Aug 29, 2023

Fixed by #2628

@dnwe dnwe closed this as completed Aug 29, 2023
ae-govau added a commit to ae-govau/sarama that referenced this issue Apr 3, 2024
@ae-govau
fix: return ErrMessageSizeTooLarge when message is too large
7f32943 
For most of this library's existence it has returned ErrMessageSizeTooLarge
when the message exceeded the configured size.

For a short period in 2019 this error was renamed (IBM#1218) but shortly
revered back (IBM#1262). Later in 2023 this error was changed to a
ConfigurationError (IBM#2628) to fix IBM#2137, however this has caused issues
with clients who rely on the previous error code being distinct from
other ConfigurationError conditions (IBM#2655).

This commit reverts to previous behaviour, and adds a test to pickup if
this changes again in the future.

Signed-off-by: Adam Eijdenberg <adam.eijdenberg@defence.gov.au>
@ae-govau ae-govau mentioned this issue Apr 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug needs-investigation Issues that require followup from maintainers
Projects
None yet
Milestone
No milestone
2 participants
@dnwe @shermanCRL