Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Issue #11351] Parallel Precise Publish Rate Limiting Fix #11372

Merged
merged 4 commits into from Jul 29, 2021
Merged

[Issue #11351] Parallel Precise Publish Rate Limiting Fix #11372

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
408030c
fixed parallel rate limiting in precise publish rate
ronfarkash Jul 19, 2021
2b9cf79
reverted from hashmap and changed function name to be more descriptive
ronfarkash Jul 23, 2021
c95209b
added small explanation on tryReleaseConnectionThrottle()
ronfarkash Jul 23, 2021
de68272
Merge branch 'master' into fix-parallel-rate-limiters-enforcement
ronfarkash Jul 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 12 additions & 2 deletions pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PrecisPublishLimiter.java
View file
Open in desktop
Expand Up @@ -63,6 +63,13 @@ public boolean isPublishRateExceeded() {
return false;
}

private void tryReleaseConnectionThrottle() {
lhotari marked this conversation as resolved.
Show resolved Hide resolved
if ((topicPublishRateLimiterOnMessage != null && topicPublishRateLimiterOnMessage.getAvailablePermits() <= 0)
|| (topicPublishRateLimiterOnByte != null && topicPublishRateLimiterOnByte.getAvailablePermits() <= 0)) {
return;
}
this.rateLimitFunction.apply();
}

@Override
public void update(Policies policies, String clusterName) {
Expand All @@ -79,10 +86,13 @@ public void update(PublishRate maxPublishRate) {
this.publishMaxByteRate = Math.max(maxPublishRate.publishThrottlingRateInByte, 0);
if (this.publishMaxMessageRate > 0) {
topicPublishRateLimiterOnMessage =
new RateLimiter(publishMaxMessageRate, 1, TimeUnit.SECONDS, rateLimitFunction);
new RateLimiter(publishMaxMessageRate, 1, TimeUnit.SECONDS,
this::tryReleaseConnectionThrottle);
}
if (this.publishMaxByteRate > 0) {
topicPublishRateLimiterOnByte = new RateLimiter(publishMaxByteRate, 1, TimeUnit.SECONDS);
topicPublishRateLimiterOnByte =
new RateLimiter(publishMaxByteRate, 1, TimeUnit.SECONDS,
this::tryReleaseConnectionThrottle);
}
} else {
this.publishMaxMessageRate = 0;
Expand Down
2 changes: 1 addition & 1 deletion pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PublishRateLimiterDisable.java
View file
Open in desktop
Expand Up @@ -59,7 +59,7 @@ public void update(PublishRate maxPublishRate) {
@Override
public boolean tryAcquire(int numbers, long bytes) {
// No-op
return false;
return true;
ronfarkash marked this conversation as resolved.
Show resolved Hide resolved
}

}
6 changes: 3 additions & 3 deletions pulsar-common/src/main/java/org/apache/pulsar/common/util/RateLimiter.java
View file
Open in desktop
Expand Up @@ -54,8 +54,8 @@ public class RateLimiter implements AutoCloseable{
private TimeUnit timeUnit;
private final boolean externalExecutor;
private ScheduledFuture<?> renewTask;
private long permits;
private long acquiredPermits;
private volatile long permits;
private volatile long acquiredPermits;
private boolean isClosed;
// permitUpdate helps to update permit-rate at runtime
private Supplier<Long> permitUpdater;
Expand Down Expand Up @@ -203,7 +203,7 @@ public synchronized boolean tryAcquire(long acquirePermit) {
*
* @return returns 0 if permits is not available
*/
public synchronized long getAvailablePermits() {
public long getAvailablePermits() {
lhotari marked this conversation as resolved.
Show resolved Hide resolved
return Math.max(0, this.permits - this.acquiredPermits);
}

Expand Down