Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka pubsub should propagate partition key to DLT #2227

Closed
ctoomey opened this issue Oct 26, 2022 · 3 comments · Fixed by #3368
Closed

Kafka pubsub should propagate partition key to DLT #2227

ctoomey opened this issue Oct 26, 2022 · 3 comments · Fixed by #3368
Labels
kind/bug Something isn't working pinned Issue does not get stale

Comments

@ctoomey
Copy link

ctoomey commented Oct 26, 2022

Expected Behavior

When a kafka pubsub subscription includes a dead letter topic, the publish to the DLT should include both the record (partition) key and value.

This is important because it's needed to support proper partitioning of the DLT as well as because some kafka consumers need the record key as well as the value for processing (ref. #2213).

Actual Behavior

Publishes to the DLT drop the record key and publish only the record value (with a null key).

Steps to Reproduce the Problem

  1. Deploy a kafka pubsub component and a corresponding pubsub subscription component that specifies a DLT
  2. Have the subscription endpoint return RETRY
  3. Publish a record with key and value to the original kafka topic
  4. Inspect the record published to the DLT and notice key is null

Release Note

RELEASE NOTE: FIX Kafka pubsub propagates partition key to DLT
@ctoomey ctoomey added the kind/bug Something isn't working label Oct 26, 2022
@dapr-bot
Copy link
Collaborator

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

@dapr-bot dapr-bot added the stale label Nov 25, 2022
@dapr-bot
Copy link
Collaborator

dapr-bot commented Dec 2, 2022

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

@dapr-bot dapr-bot closed this as completed Dec 2, 2022
@yaron2 yaron2 reopened this Dec 2, 2022
@dapr-bot
Copy link
Collaborator

dapr-bot commented Dec 9, 2022

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.

@dapr-bot dapr-bot closed this as completed Dec 9, 2022
@mukundansundar mukundansundar reopened this Dec 9, 2022
@mukundansundar mukundansundar added pinned Issue does not get stale and removed stale labels Dec 9, 2022
@denisbchrsk denisbchrsk mentioned this issue Feb 28, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working pinned Issue does not get stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Kafka PubSub: Propagate partition key to DLT denisbchrsk/components-contrib
4 participants
@ctoomey @yaron2 @dapr-bot @mukundansundar