[ISSUE #4897] Implement retry consuming based on pulsar message middleware #4898
Conversation
…y-based-on-pulsar-message-middleware
| .enableRetry(true) | ||
| .deadLetterPolicy(DeadLetterPolicy.builder() | ||
| .deadLetterTopic(dlqTopic) | ||
| .retryLetterTopic(retryTopic) |
There was a problem hiding this comment.
You have defined the retry topic here, why do you still need to manually send the message to the retry topic in PulsarRetryStrategyImpl? Doesn't Pulsar automatically send failed messages to the retry topic?
| } catch (Exception e) { | ||
| ackConsumer.negativeAcknowledge(msg); | ||
| try { | ||
| ackConsumer.reconsumeLater(msg, 5, TimeUnit.SECONDS); |
There was a problem hiding this comment.
For negativeAcknowledge() and reconsumeLater(), the former will automatically re-consume messages from the retry topic retryTopic, while the latter will re-consume messages from the current topic subTopic. Why do you need to use both? Have I misunderstood?
There was a problem hiding this comment.
The negativeAcknowledge() method enables the redelivery of failed messages by sending them back to the main queue with the main topic. The reconsumeLater() method, on the other hand, handles retries by moving the failed messages to a special retry topic stored in a retry queue, where they will be reprocessed after a specified delay.
There was a problem hiding this comment.
Thanks for the correction. So why? There are two retries here, and the default retry of EventMesh. So why does Pulsar's retry need to be set to three times instead of just once?
There was a problem hiding this comment.
Compared with negative acknowledgment, retry letter topic is more suitable for messages that require a large number of retries with a configurable retry interval.The messages in the retry letter topic are persisted to BookKeeper, while messages that need to be retried due to negative acknowledgment are cached on the client side.Negative acks and retry/DLQ do work together
There was a problem hiding this comment.
Negative acks and retry/DLQ do work together
Are you sure it's a collaboration and not duplicate consumption? If you are sure, could you explain this: how can it ensure that only one of the mechanisms will be triggered for retries when there are failed messages, given that both mechanisms exist simultaneously?
There was a problem hiding this comment.
Sorry, I had a misunderstanding. The negative acknowledgment will cause duplicate consumption with active retry. I have fixed it.
…tegy-based-on-pulsar-message-middleware' into apache#4556-Implement-retry-strategy-based-on-pulsar-message-middleware
There was a problem hiding this comment.
I found that the original design of the retry module seems to have some problems, but I'm not sure yet. I need to close issue #4556 first. However, your PR can serve as the implementation of a separate issue and will not be affected.
|
Based on my current understanding of Pulsar, I have completed the review to the best of my ability. As I said in your last PR, I am not very familiar with Pulsar, so I can't guarantee the quality of my review this time. Please forgive my lack of technical expertise. Reviewers should be responsible for the first "approved" of each PR, so please wait for further review from more members of the community. |
pandaapo
changed the title
pandaapo
changed the title
Fixes #4897
Motivation
This implementation will add retry consuming on Pulsar message middleware
Modifications
Implemented the code for supporting retry based on Pulsar
Documentation