Sending over-simple DeduplicationId to FIFO type AWS SNS dropped "resolved" message

[tyrken]

What did you do?

Configured an AWS SNS FIFO topic as a receiver to allow downstream notifications via an external system.

What did you expect to see?

SNS notifications (via an attached SQS) from alerts as they both fire and resolve, as we used send_resolved: true

What did you see instead? Under which circumstances?

This worked for most situations, but if an alert resolved within a few minutes of firing we'd get the original alert but no "resolved" message.

Environment

• System information:

Linux 5.10.205-195.804.amzn2.x86_64 x86_64

• Alertmanager version:

0.26.0

• Alertmanager configuration file:

...
    sns_configs:
      - sigv4:
          region: eu-west-1
        topic_arn: arn:aws:sns:eu-west-1:myaccount:mysns.fifo
        # If the SNS/message looks like JSON the SNS/subject must be blank or get 400 error
        subject: ""
        message: '{{ template "extras.mytemplate" .}}'
        attributes:
          # SNS attributes are forced to strings, but must never be empty
          source: alertmanager
          status: "{{.Status}}"
          num_alerts: "{{len .Alerts}}"

• Logs:

2024-01-30T12:39:24.824514000Z {"caller":"sns.go:94","integration":"sns","level":"debug","message_id":"cc112344-7d61-4432-9732-8ff6435d29a8","msg":"SNS message successfully published","sequence number":"10000000000006784000","ts":"2024-01-30T12:39:24.821Z"}
...
# 3 mins pass before it tries sending the non-identical "Resolved" message - the Status field is included in "mytemplate" so body content is different
...
2024-01-30T12:42:24.615541000Z {"caller":"sns.go:94","integration":"sns","level":"debug","message_id":"cc112344-7d61-4432-9732-8ff6435d29a8","msg":"SNS message successfully published","sequence number":"10000000000006784000","ts":"2024-01-30T12:42:24.615Z"}

Overall I think this is exactly the situation worried about in #2559 (comment) . As we set a DeduplicationId in

alertmanager/notify/sns/sns.go

Lines 158 to 163 in 355c6df

	key, err := notify.ExtractGroupKey(ctx)
	if err != nil {
	return nil, err
	}
	publishInput.SetMessageDeduplicationId(key.Hash())
	publishInput.SetMessageGroupId(key.Hash())

then SNS trusts it to dedupe - but the Firing and Resolved messages will have the same Group Key and lead to dropping the second message. If you left DeduplicationId blank then AWS would use a hash of the content to calc a DeduplicationId - which I think is better behaviour.