New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PreciseTopicPublishRateLimiterEnable doesn't always work. #10382
Comments
@galrose thanks for reporting . When reproducing, I needed to also specify a message limit to the publish rate limit. I was able to reproduce the issue. When the amount of outstanding messages is high the rate limiting is very inconsistent. For example:
The current algorithm in the rate limiter seems to have it's limitations. It seems to work in a way where it switches the "auto read" to false for the Netty channel to cause backpressure when the rate limit is reached. However, it seems that the rate limit will reset once per second. This has the consequence that buffered content will get resumed. |
Yes you are correct my bad, you can also do just the -m for message limitation and you can reproduce it like that as well. |
Describe the bug
When using the pulsar-perf and writing messages either in large batches (>100) or large number of outstanding messages (>100).
When limiting either the -bm or -o to a small number around 5, preferably 1 it works perfectly.
The easiest way to check is not setting the -bm or -o in pulsar-perf.
Slack thread: https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1619419736156600?thread_ts=1619419736.156600&cid=C5Z4T36F7
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I expect the limit to be precise no matter the batch or outstanding messages.
Desktop (please complete the following information):
The text was updated successfully, but these errors were encountered: