PIP 3: Message dispatch throttling
- Status: Implemented
- Author: Rajan Dhabalia
- Pull Request: #634
Producers and consumers can interact with broker on high volumes of data. This can monopolize broker resources and cause network saturation, which can have adverse effects on other topics owned by that broker. Throttling based on resource consumption can protect against these issues and it can play important role for large multi-tenant clusters where a small set of clients using high volumes of data can easily degrade performance of other clients.
Sometimes, namespace with large message backlog can continuously drain messages with multiple connected consumers, which can easily over utilize broker’s bandwidth. Therefore, we can configure dispatch-rate for that namespace which will enforce throttling while dispatching messages to the consumers. Throttling limit can be configured at namespace level by defining dispatch-rate threshold which will apply to each of the topic under that namespace.
By default, broker does not throttle message dispatching for any namespace unless cluster is configured with default throttling-limit or namespace has been configured as a part of namespace policy in a cluster. We can configure throttling limit for a specific namespace using an admin-api.
Broker already has capability to throttle number of publish messages by reading only fixed number of in flight messages per client connection, which can protect broker if client tries to publish significantly large number of messages.
After configuring dispatch rate-limit, each topic can dispatch with in that rate-limit . If a topic is configured with rate-limiting of 10 message per second then broker dispatches only 10 messages per second for that topic.
Message-rate limit applies across all the subscriptions for a loaded topic. Therefore, broker initializes rate-limiter with configured number of limit which will be shared across all the subscriptions of the topic while dispatching messages. So, subscription reads available messages only if rate-limiter has permits available else subscription backs off and schedule next read after a second.
If subscription doesn’t have backlog and it has already caught up then it reads published messages from the broker’s cache and it doesn’t have to read from the bookie. Therefore, broker doesn’t apply message-rate limiting for the subscribers which are already caught up with producers and consuming messages without building a backlog.
Throttling limit can be defined in two ways:
Message-rate: Message dispatching can be throttled by total number of message per duration. If message-rate is configured 100/second then broker dispatches only 100 messages per second.
Byte-rate: Message dispatching can be throttled by total number of bytes per duration. If byte-rate is configured 1MB/second then broker dispatches messages with total size of 1 MB per second.
If both the limits are configured for a topic then broker applies both the limit and throttles based on whichever threshold reaches first.
By default, broker does not throttle message dispatching for any of the topics. However, if we want to uniformly distribute broker’s resources across all the topics then, we can configure default message-rate at cluster level and it will be immediately effective dynamically after configuring it . This default dispatch-rate in a cluster will apply to all the topics of all the brokers serving in that cluster. However, namespace with already configured throttling will override cluster’s default limit while dispatching messages for all the topics under that namespace.
Following configuration forces every broker in the cluster to dispatch only 1000 messages per second for each topic served by the broker.
Pulsar-admin brokers update-dynamic-config --config dispatchThrottlingRatePerTopicInMsg --value 1000Following configuration forces every broker in the cluster to dispatch only 1000 bytes per second for each topic served by the broker.
Pulsar-admin brokers update-dynamic-config --config dispatchThrottlingRatePerTopicInByte --value 1000By default, value of both the configurations is 0 which disables throttling in that cluster.
We can always override the default throttling-limit for namespace that need a higher or lower dispatch-rate. We can configure throttling-limit for all the topics under a specific namespace and it will be immediately effective dynamically after configuring it. Also, throttling-limit for a namespace will be configured per cluster that gives flexibility to configure specific dispatch-rate for the namespace in each cluster. Dispatch-rate configuration for each namespace will be stored under /admin/policiesinto global-zookeeper.
Following configuration forces broker to dispatch only 1000 messages per second for each topic under the given namespace.
pulsar-admin namespaces <property/cluster/namespace> set-dispatch-rate --msg-dispatch-rate 1000
Following configuration forces broker to dispatch only 1000 bytes per second for each topic under the given namespace.
pulsar-admin namespaces <property/cluster/namespace> set-dispatch-rate --byte-dispatch-rate 1000
By default, value of both the configurations is 0 which disables dispatch-throttling for the namespace.
In above approach, broker does message dispatching throttling per namespace. However, there could be a high possibility that specific subscriber of the topic has a large backlog and over consuming bandwidth of the broker. Therefore, other topics or subscribers under the same namespace can be impacted and suffer due to namespace level throttling. In that case, we can provide an option to configure subscriber level throttling by storing subscriber configuration (under /managed-ledger/property/cluster/ns/persistent/topic/subscriber/configuration) into zookeeper. However, it can create an administrative complexity to manage configurations for every topic and subscriber.
As described in #402, we can define rate limiting policy that maps to the list of regex which can match to the topics for which we want to define throttling. This approach can give more granular control by throttling on topic level. However, on a long run it might be complex or difficult to manage rate limiting policies for large number of topics.