Pulsar is creating a unlimited number topics with name _transaction_pending_ack-__pending_ack_state__transaction_pending_ack

[eolivelli]

Describe the bug

In my application I see that Pulsar is creating many topics, appending "_transaction_pending_ack-__pending_ack_state__transaction_pending_ack" to the previous topic name.
I see this "topic name"

persistent://public/default/MY_TOPIC-8975a6f5-be96-4c03-b5b9-02ad570dd1f9__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack-__pending_ack_state__transaction_pending_ack

Each topic has one subscription '__pending_ack_state'

To Reproduce
Create a Consumer with Transactions and EnableAckReceipt, use it

Expected behavior
Only at topic should be created

** Context
Pulsar version 2.8.1

[eolivelli]

@congbobo184 @codelipenghui FYI

[eolivelli]

basically you start with persistent://public/default/MY_TOPIC
then Pulsar creates a new system topic persistent://public/default/MY_TOPIC__transaction_pending_ack (this happens for every topic when you enable the Transaction Coordinator)

then Pulsar creates a new topic persistent://public/default/MY_TOPIC__transaction_pending_ack-__pending_ack_state
this event triggers the creation of persistent://public/default/MY_TOPIC__transaction_pending_ack-__pending_ack_state__transaction_pending_ack

and so on...

[eolivelli]

I believe that the problem is because for each topic we create a new topic with this suffix __transaction_pending_ack
and one subscription named __pending_ack_state
but for each subscription we create a new topic named TOPIC_NAME + subscriptionName + __transaction_pending_ack
so TOPIC_NAME + __pending_ack_state + __transaction_pending_ack

then again TOPIC_NAME becomes TOPIC_NAME + __pending_ack_state + __transaction_pending_ack and the process proceeds

See here:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java

Line 53 in be4f9a7

.getTransactionPendingAckStoreSuffix(originPersistentTopic.getName(), subscription.getName());

[codelipenghui]

@congbobo184 Could you please check this issue?

[eolivelli]

it looks like it is a regression in 2.8.1, I didn't see that behaviour in 2.8.0

[eolivelli]

I have found a "workaround", basically the thing that triggers this problem is a tool that I have that lists all the topics and for each topic calls getSubscriptions:

List<String> allTopics = pulsarAdmin.topics().getList(systemNamespace);
for (String topic : allTopics) {
         
          log.info("Scanning topic {}", topic);
          List<String> subscriptions = pulsarAdmin.topics().getSubscriptions(topic);
}

if you run this loop many times you will see this problem.

so my understanding is that getSubscriptions is creating those system topics even for system subscriptions

[congbobo184]

this is a bug, I will fix it.

[eaba]

@eolivelli thanks for capturing this issue. In my case such topics were in thousands and this affected Pattern Consumer. I actually thought it was error from my end but now this issue has been opened and closed, I am relieved.

[eolivelli]

@eaba are you using transactions in productions?

If so please take care of testing the upgrade from 2.8 to 2.9.
There are a few changes in the wire protocol for transactions.
Please report any issue

[eaba]

@eolivelli thanks for the heads up.