Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] How to solve or optimize this problem [Client memory buffer is full] #17360

Open
2 tasks done
rogueVpedro opened this issue Aug 31, 2022 · 7 comments
Open
2 tasks done
Labels
Stale type/bug The PR fixed a bug or issue reported a bug

Comments

@rogueVpedro
Copy link

Search before asking

  • I searched in the issues and found nothing similar.

Version

2.10.1 master

Minimal reproduce step

When we call the client for a long time, we find that the client is down and cannot send messages.
Then we got an exception while calling the client producer to send messages. (Maybe the frequency of use is too high)
screen_shot_1661910571587
Before asking, we have checked the relevant issues #15724 and #15691,but we still don't know how to avoid this situation.
And this is our process of creating producers
screen_shot_1661911158535
For the configuration of these two parameters(maxPendingMessages and memoryLimit), we set the default values . Btw,The JVM memory of the producer client is 10g and our server memory is 24g.When a problem occurs, out of heap memory is sufficient.
So,how to solve or optimize this problem [Client memory buffer is full]

What did you expect to see?

Avoid this problem, or recover to the original normal situation when the client hangs up.

What did you see instead?

Failing

Anything else?

We sincerely hope to solve this problem,

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@Technoboy-
Copy link
Contributor

The default memoryLimit is 64M.
Try to set a larger value.

PulsarClient.builder()
                    .memoryLimit(xxx, SizeUnit.MEGA_BYTES)

@gaozhangmin
Copy link
Contributor

If you encountered send error, like timeout, there were multiple bug about client memory release, please reference
#16837
#16915
#16957
#16958
#16971

@rogueVpedro
Copy link
Author

默认内存限制为 64M。 尝试设置更大的值。

PulsarClient.builder()
                    .memoryLimit(xxx, SizeUnit.MEGA_BYTES)

In fact, the size of each message is controlled within 64M. We judge the size of each message before sending it to the producer queue. But still found this problem again

@rogueVpedro
Copy link
Author

If you encountered send error, like timeout, there were multiple bug about client memory release, please reference #16837 #16915 #16957 #16958 #16971

OK, I will refer to the solutions to these bugs, but I am not sure whether they are useful.

@github-actions
Copy link

github-actions bot commented Oct 2, 2022

The issue had no activity for 30 days, mark with Stale label.

@github-actions github-actions bot added the Stale label Oct 2, 2022
@Jason918
Copy link
Contributor

Jason918 commented Oct 2, 2022

In fact, the size of each message is controlled within 64M. We judge the size of each message before sending it to the producer queue. But still found this problem again

@rogueVpedro FYI: 64MB is the total memory limit, not for each message.

@github-actions github-actions bot removed the Stale label Oct 3, 2022
@github-actions
Copy link

github-actions bot commented Nov 2, 2022

The issue had no activity for 30 days, mark with Stale label.

@github-actions github-actions bot added the Stale label Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Stale type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

No branches or pull requests

4 participants