Topic Compaction Strategy

[marekczajkowski]

It would be nice to be able to set a compaction strategy on topic/namespace level. Something similar to what it's available for TableView where we can pass the compactionStrategy class name to load TopicCompactionStrategy but on the broker.

Such a change would make topic compaction more flexible. Let's think of an application that sets eventTime on messages when a message is produced. These messages can be sent one by one with time T1,T2...TN. Despite the fact that messages are sent in order they may be put to topic in different order. We would like to compact the topic based on eventTime to have the last sent messages on the compacted topic instead of the last received.

[mattisonchao]

cc @coderzc

[michalcukierman]

Should be predefined strategies or is there a mechanism to extend a classpath and add custom strategies?
I currently see the need of strategies working on:

• publicationTime
• eventTime
• order (as it's now)

Probably publication time and event time could be used in a single strategy.

[michalcukierman]

@mattisonchao @coderzc
Do you have any thoughts on the topic. The issue is related to using Functions composed into Meshes. If a function uses TableView, wrong events may be compacted.

[michalcukierman]

Hey, any update on this @mattisonchao , @coderzc ?

[lhotari]

cc @heesung-sn , any comment on // TODO: Create compaction strategy from topic policy when exposing strategic compaction to users. that you have added in PersistentTopic.java ? https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L241 ?

[heesung-sn]

Currently, strategic compaction is only used in one system topic for the bundle state channel. If the community wants to expose this strategic compaction for all topics, I think it makes sense to add the strategic compaction option in the topic policy. I guess this might require a quick discussion/vote from the Pulsar community.