Skip to content
This repository has been archived by the owner on Apr 1, 2024. It is now read-only.

ISSUE-11351: Fix precise publish rate limiting #2788

Open
sijie opened this issue Jul 17, 2021 · 0 comments
Open

ISSUE-11351: Fix precise publish rate limiting #2788

sijie opened this issue Jul 17, 2021 · 0 comments

Comments

@sijie
Copy link
Member

sijie commented Jul 17, 2021

Original Issue: apache#11351


Is your enhancement request related to a problem? Please describe.
Currently, precise publish rate limiting doesn't work as expected and that's because multiple reasons

  1. Renew function called every second and resets reading from socket every second no matter how many messages produced in the last second (FixedWindow Algorithm)
  2. The FixedWindow algorithm does not fit here because a permit is not 1:1 (multiple messages in batch, bytes per second)
  3. Rate limit function passed only to the msg/s rate limiter (and that's in order to avoid calling it twice)
  4. The first message of any new producer that connects to the broker will get in

Describe the solution you'd like
1 + 2. in order to fix that, I believe we should implement another RateLimiter using the LeakingBucket Algorithm along with FixedWindow
3. we should pass a rate limit function that depends on the state of both of the rate limiters
4. I don't really have an idea how to implement a fix for it would love to hear ur opinions

Additional context
The main idea is thanks to apache#8611 (comment) @lhotari

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant