Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow SNSSQS MessageGroupId to be set programatically via pubsub metadata #3361

Open
Jossec101 opened this issue Feb 22, 2024 · 2 comments
Open
Labels
help wanted Extra attention is needed kind/enhancement New feature or request
Milestone

Comments

@Jossec101
Copy link
Contributor

Describe the feature

We have a business logic which requires to be able to take advantage of the MessageGroupId to be set programatically as we want to do FIFO parallel consumption but sequentially based on the same MessageGroupId groups.

I guess that adding a flag in the metadata and modifying something like FifoMessageGroupId key and checking if set, use it as a the result var for the function could make it.

We can do the PR work ;)

func (s *snsSqs) getMessageGroupID(req *pubsub.PublishRequest) *string {
if len(s.metadata.FifoMessageGroupID) > 0 {
return &s.metadata.FifoMessageGroupID
}
// each daprd, of a given PubSub, of a given publisher application publishes to a message group ID of its own.
// for example: for a daprd serving the SNS/SQS Pubsub component we generate a unique id -> A; that component serves on behalf
// of a given PubSub deployment name B, and component A publishes to SNS on behalf of a dapr application named C (effectively to topic C).
// therefore the created message group ID for publishing messages in the aforementioned setup is "A:B:C".
fifoMessageGroupID := fmt.Sprintf("%s:%s:%s", s.id, req.PubsubName, req.Topic)
return &fifoMessageGroupID

The code could look like this:

func (s *snsSqs) getMessageGroupID(req *pubsub.PublishRequest) *string {
	// if the request has provided a messageGroupID in the metadata, use it.
	if req.Metadata["messageGroupID"] != "" {
		fifoMessageGroupID := req.Metadata["messageGroupID"]
		return &fifoMessageGroupID
	}

	if len(s.metadata.FifoMessageGroupID) > 0 {
		return &s.metadata.FifoMessageGroupID
	}

	// each daprd, of a given PubSub, of a given publisher application publishes to a message group ID of its own.
	// for example: for a daprd serving the SNS/SQS Pubsub component we generate a unique id -> A; that component serves on behalf
	// of a given PubSub deployment name B, and component A publishes to SNS on behalf of a dapr application named C (effectively to topic C).
	// therefore the created message group ID for publishing messages in the aforementioned setup is "A:B:C".
	fifoMessageGroupID := fmt.Sprintf("%s:%s:%s", s.id, req.PubsubName, req.Topic)
	return &fifoMessageGroupID
}

Release Note

RELEASE NOTE: ADD SNSSQS FifoMessagroupId can be set programatically via pubsub req metadata.

@Jossec101 Jossec101 added the kind/enhancement New feature or request label Feb 22, 2024
@berndverst berndverst added this to the v1.14 milestone Feb 22, 2024
@berndverst berndverst added the help wanted Extra attention is needed label Feb 22, 2024
@berndverst
Copy link
Member

@Jossec101 sounds good - can you comment here with /assign please? This of course cannot be released until Dapr 1.14

@berndverst
Copy link
Member

One complication factor: You also should cover the new functionality in our integration test (certification test) here: https://github.com/dapr/components-contrib/tree/main/tests/certification/pubsub/aws/snssqs

Writing the test and verifying that everything works (e.g. calling the AWS SDK manually to check it worked on the server) is the hardest part here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed kind/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants