Home
brod_consumer is essentially a poller, constantly asking for more data from the kafka node which is a leader for the given partition.
By subscribing to brod_consumer a process should receive the polled message sets (not individual messages) into its mailbox.
In brod, we have so far implemented two different subscribers (brod_topic_subscriber and brod_group_subscriber), hopefully to fulfil most of the common use cases.
For maximum flexibility, an application may implement its own per-partition subscriber.
Below diagrams illustrate how subscriber processes might be organised
This gives the best flexibility as the per-partition subscribers work directly with per-partition pollers.
The messages are delivered to subscribers in message sets (batches), not individual messages, (however the subscribers are allowed to ack individual offsets).
A topic subscriber provides the easiest way to receive and process messages from ALL partitions of a given topic.
See brod_demo_cg_collector and brod_demo_topic_subscriber for example.
Users may choose to implement the brod_topic_subscriber behaviour callbacks in a module,
or simply provide an anonymous callback function to have the individual messages processed.
Similar to topic subscriber the brod_group_subscriber behaviour callbacks should be implemented to process individual messages.
See brod_demo_group_subscriber_koc and brod_demo_group_subscriber_koc for example.
A group subscriber is started by giving a set of topics, some (maybe none, or maybe all) of the partitions in the topic set should be assigned to it, and the subscriber should subscribe to ALL the assigned partitions.
Users may choose to implement the brod_group_member behaviour (callbacks for brod_group_coordinator) for a different group subscriber (e.g. spawn one subscriber per partition). see brucke for example.