This repository has been archived by the owner on Jan 8, 2020. It is now read-only.
Round-robin partition assignments across multiple topics #216
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is closer to the behavior of Kafka's Java consumer, as described
here:
https://github.com/apache/kafka/blob/15bc405/clients/src/main/java/org/apache/kafka/clients/consumer/RoundRobinAssignor.java#L31-L55
This is valuable in situations where you have highly disparate
partition counts. Given a few group members subscribed to many topics,
only some of which have more than one partition, the first member
will receive far more assignments than the others.
As an example, I've created topics
t0
..t9
with a single partition, thent10
andt11
with 4 partitions. I booted two consumers compiled againstsarama-cluster master in the same group and printed out their assignments
once both had joined:
Using this branch instead:
So far, I've tried not to change too much: the existing test cases aren't
actually different, just uglier because of the more complicated API.
The additional test cases introduced match the scenario in the comment
linked above.
I'd actually like to make a bigger change, but didn't want to start down that path
without getting some feedback first. It would be nice to instead expose the
assignment strategy implementation as an interface, so users could provide
the logic. Something like:
This would enable users to do things like try implementing the StickyAssignor,
which is a ton of code and would probably be twice as much in Go, without needing to
fork or get the code upstream before it's been put through its paces.
Current implementation here could become this without too much additional effort,
just gotta work out what the most ergonomic types would be for the inputs and outputs
to the assignor.