Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS SNS receiver #2615

Merged
merged 35 commits into from Jul 26, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
19e74f9
WIP - SNS receiver
Jun 10, 2021
5dcf4f5
ARN Auth start
Jun 11, 2021
74d1527
Add support for role arn, truncation, dedupe key and env auth
Jun 11, 2021
009f8b1
Use 1024 rather than 1000 for KB size, fix target arn, handle large S…
Jun 14, 2021
72d63a5
Remove isFifo config option; use template strings; use retier; other …
Jun 14, 2021
6519c39
Add some tests for sns receiver
Jun 15, 2021
af8406a
Check error type before unpacking awserr.requestFailure
Jun 15, 2021
68fa1bf
Add string length check to fifo check
Jun 15, 2021
b509a5b
Add subject template for subject field. Better check for supplied cre…
Jun 15, 2021
8d3b1b5
Add config docs
Jun 15, 2021
889fa96
Remove isFifoTopic test
Jun 15, 2021
c48b54b
Fix gosmpl linter issues
Jun 16, 2021
3a63cc2
Updated assets
pracucci Jun 16, 2021
6ada9a6
Cache fifo bool in the notifier
Jun 16, 2021
756cdda
Fix for golangci-lint warning
Jun 16, 2021
9d37d6c
More gofmt fixes
Jun 16, 2021
3446b35
Code review fixes: copy attributes, truncate all the messages, fix lo…
Jun 16, 2021
a56305a
Fix spacing from removing default api version
Jun 16, 2021
d4ff90b
Add missing template for aws region
Jun 16, 2021
b9b53f1
Code review fixes
Jun 17, 2021
63f9082
Fix docs spacing
Jun 17, 2021
4ebcaf9
Merge remote-tracking branch 'upstream/master' into sns-reciever
Jun 17, 2021
8911051
Make API URL optional, clear up credential logic
Jun 21, 2021
dfb4d1f
Fix linter error
Jun 21, 2021
9ff4ac3
Create new session if needed to get STS Creds
Jun 21, 2021
30a83f7
Use supplied user creds when creating an STS client
Jun 22, 2021
bd82f70
Fix spacing for client config
Jun 22, 2021
25e6d4e
Add common/sigv4 with the sigv4 config
Jun 23, 2021
208bed6
Update config docs to clarify fifo SNS deduplication strategy. Remove…
Jun 28, 2021
1322abd
Remove unused checkTopicFifoAttribute function
Jun 28, 2021
077b20d
Add error check when creating sns session
Jul 1, 2021
7ecb6bc
Check Error in unit test and clean up docs
Jul 6, 2021
4c2a5f1
Add sigv4 as a global config option
Jul 7, 2021
51b9368
Revert "Add sigv4 as a global config option"
Jul 9, 2021
a1260af
Break notify into submethods to create the session then create the pu…
Jul 9, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions cmd/alertmanager/main.go
Expand Up @@ -31,6 +31,7 @@ import (
"github.com/go-kit/kit/log"
"github.com/go-kit/kit/log/level"
"github.com/pkg/errors"
"github.com/prometheus/alertmanager/notify/sns"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/prometheus/common/model"
Expand Down Expand Up @@ -165,6 +166,9 @@ func buildReceiverIntegrations(nc *config.Receiver, tmpl *template.Template, log
for i, c := range nc.PushoverConfigs {
add("pushover", i, c, func(l log.Logger) (notify.Notifier, error) { return pushover.New(c, tmpl, l) })
}
for i, c := range nc.SNSConfigs {
add("sns", i, c, func(l log.Logger) (notify.Notifier, error) { return sns.New(c, tmpl, l) })
}
if errs.Len() > 0 {
return nil, &errs
}
Expand Down
9 changes: 9 additions & 0 deletions config/config.go
Expand Up @@ -242,6 +242,9 @@ func resolveFilepaths(baseDir string, cfg *Config) {
for _, cfg := range receiver.WechatConfigs {
cfg.HTTPConfig.SetDirectory(baseDir)
}
for _, cfg := range receiver.SNSConfigs {
cfg.HTTPConfig.SetDirectory(baseDir)
}
}
}

Expand Down Expand Up @@ -447,6 +450,11 @@ func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
voc.APIKey = c.Global.VictorOpsAPIKey
}
}
for _, sns := range rcv.SNSConfigs {
if sns.HTTPConfig == nil {
sns.HTTPConfig = c.Global.HTTPConfig
}
}
names[rcv.Name] = struct{}{}
}

Expand Down Expand Up @@ -784,6 +792,7 @@ type Receiver struct {
WechatConfigs []*WechatConfig `yaml:"wechat_configs,omitempty" json:"wechat_configs,omitempty"`
PushoverConfigs []*PushoverConfig `yaml:"pushover_configs,omitempty" json:"pushover_configs,omitempty"`
VictorOpsConfigs []*VictorOpsConfig `yaml:"victorops_configs,omitempty" json:"victorops_configs,omitempty"`
SNSConfigs []*SNSConfig `yaml:"sns_configs,omitempty" json:"sns_configs,omitempty"`
}

// UnmarshalYAML implements the yaml.Unmarshaler interface for Receiver.
Expand Down
53 changes: 53 additions & 0 deletions config/notifiers.go
Expand Up @@ -127,6 +127,16 @@ var (
Expire: duration(1 * time.Hour),
HTML: false,
}

// DefaultSNSConfig defines default values for SNS configurations.
DefaultSNSConfig = SNSConfig{
NotifierConfig: NotifierConfig{
VSendResolved: true,
},
APIVersion: "sns.default.api_version",
treid314 marked this conversation as resolved.
Show resolved Hide resolved
Message: `{{ template "sns.default.message" . }}`,
IsFIFOTopic: false,
}
)

// NotifierConfig contains base options common across all notifier configurations.
Expand Down Expand Up @@ -579,3 +589,46 @@ func (c *PushoverConfig) UnmarshalYAML(unmarshal func(interface{}) error) error
}
return nil
}

// TODO: Move to common?
treid314 marked this conversation as resolved.
Show resolved Hide resolved

// SigV4Config is the configuration for signing remote write requests with
// AWS's SigV4 verification process. Empty values will be retrieved using the
// AWS default credentials chain.
type SigV4Config struct {
Region string `yaml:"region,omitempty"`
AccessKey string `yaml:"access_key,omitempty"`
SecretKey Secret `yaml:"secret_key,omitempty"`
Profile string `yaml:"profile,omitempty"`
RoleARN string `yaml:"role_arn,omitempty"`
}

type SNSConfig struct {
NotifierConfig `yaml:",inline" json:",inline"`

HTTPConfig *commoncfg.HTTPClientConfig `yaml:"http_config,omitempty" json:"http_config,omitempty"`

APIUrl string `yaml:"api_url" json:"api_url"`
treid314 marked this conversation as resolved.
Show resolved Hide resolved
APIVersion string `yaml:"api_version,omitempty" json:"api_version,omitempty"`
Sigv4 SigV4Config `yaml:"sigv4" json:"sigv4"`
TopicARN string `yaml:"topic_arn,omitempty" json:"topic_arn,omitempty"`
PhoneNumber string `yaml:"phone_number,omitempty" json:"phone_number,omitempty"`
TargetARN string `yaml:"target_arn,omitempty" json:"target_arn,omitempty"`
IsFIFOTopic bool `yaml:"is_fifo_topic,omitempty" json:"is_fifo_topic,omitempty"`
treid314 marked this conversation as resolved.
Show resolved Hide resolved
Subject string `yaml:"subject,omitempty" json:"subject,omitempty"`
Message string `yaml:"message,omitempty" json:"message,omitempty"`
Attributes map[string]string `yaml:"attributes,omitempty" json:"attributes,omitempty"`
}

// UnmarshalYAML implements the yaml.Unmarshaler interface.
func (c *SNSConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
*c = DefaultSNSConfig
type plain SNSConfig
if err := unmarshal((*plain)(c)); err != nil {
return err
}
if c.TargetARN == "" && c.TopicARN == "" && c.PhoneNumber == "" {
treid314 marked this conversation as resolved.
Show resolved Hide resolved
return fmt.Errorf("must provide either a Target ARN, Topic ARN, or Phone Number for SNS config")
}
return nil
}
16 changes: 16 additions & 0 deletions config/testdata/conf.sns-test.yml
@@ -0,0 +1,16 @@
route:
treid314 marked this conversation as resolved.
Show resolved Hide resolved
receiver: 'sns-api-notifications'
group_by: [alertname]

receivers:
- name: 'sns-api-notifications'
sns_configs:
- api_url: https://sns.us-east-2.amazonaws.com
topic_arn: arn:aws:sns:us-east-2:123456789012:My-Topic
is_fifo_topic: true
sigv4:
region: us-east-2
access_key: access_key
secret_key: secret_ket
attributes:
severity: Sev2
1 change: 1 addition & 0 deletions go.mod
Expand Up @@ -2,6 +2,7 @@ module github.com/prometheus/alertmanager

require (
github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15
github.com/aws/aws-sdk-go v1.38.35
github.com/cenkalti/backoff/v4 v4.1.0
github.com/cespare/xxhash v1.1.0
github.com/go-kit/kit v0.10.0
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Expand Up @@ -39,6 +39,8 @@ github.com/asaskevich/govalidator v0.0.0-20200907205600-7a23bdc65eef/go.mod h1:W
github.com/aws/aws-lambda-go v1.13.3/go.mod h1:4UKl9IzQMoD+QF79YdCuzCwp8VbmG4VAQwij/eHl5CU=
github.com/aws/aws-sdk-go v1.27.0/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
github.com/aws/aws-sdk-go v1.38.35 h1:7AlAO0FC+8nFjxiGKEmq0QLpiA8/XFr6eIxgRTwkdTg=
github.com/aws/aws-sdk-go v1.38.35/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
Expand Down Expand Up @@ -301,7 +303,9 @@ github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod
github.com/jessevdk/go-flags v1.5.0 h1:1jKYvbxEjfUl0fmqTCOfonvskHHXMjBySTLW4y9LFvc=
github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
Expand Down
1 change: 1 addition & 0 deletions notify/notify.go
Expand Up @@ -277,6 +277,7 @@ func NewMetrics(r prometheus.Registerer) *Metrics {
"opsgenie",
"webhook",
"victorops",
"sns",
} {
m.numNotifications.WithLabelValues(integration)
m.numTotalFailedNotifications.WithLabelValues(integration)
Expand Down
171 changes: 171 additions & 0 deletions notify/sns/sns.go
@@ -0,0 +1,171 @@
// Copyright 2021 Prometheus Team
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package sns

import (
"context"
"fmt"
"net/http"
"unicode/utf8"

"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/stscreds"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/sns"
"github.com/go-kit/kit/log"
"github.com/prometheus/alertmanager/config"
"github.com/prometheus/alertmanager/notify"
"github.com/prometheus/alertmanager/template"
"github.com/prometheus/alertmanager/types"
commoncfg "github.com/prometheus/common/config"
)

// Notifier implements a Notifier for SNS notifications.
type Notifier struct {
conf *config.SNSConfig
tmpl *template.Template
logger log.Logger
client *http.Client
retrier *notify.Retrier
treid314 marked this conversation as resolved.
Show resolved Hide resolved
}

func (n Notifier) Notify(ctx context.Context, alert ...*types.Alert) (bool, error) {
credentials := credentials.NewStaticCredentials(n.conf.Sigv4.AccessKey, string(n.conf.Sigv4.SecretKey), "")
treid314 marked this conversation as resolved.
Show resolved Hide resolved
if n.conf.Sigv4.AccessKey == "" {
credentials = nil
}
treid314 marked this conversation as resolved.
Show resolved Hide resolved

sess, err := session.NewSessionWithOptions(session.Options{
treid314 marked this conversation as resolved.
Show resolved Hide resolved
Config: aws.Config{
Region: aws.String(n.conf.Sigv4.Region),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving the comment here, but applies to many config options. If you look at the specs in #2559, you can see many config options should actually be tmpl_string. The string is not evaluated as a template here (correct me if I'm wrong), but I think you should. As an example, look at tmplText() usage in the Slack integration.

Credentials: credentials,
Endpoint: aws.String(n.conf.APIUrl),
},
Profile: n.conf.Sigv4.Profile,
})

if _, err := sess.Config.Credentials.Get(); err != nil {
return false, fmt.Errorf("could not get SigV4 credentials: %w", err)
}
treid314 marked this conversation as resolved.
Show resolved Hide resolved

if n.conf.Sigv4.RoleARN != "" {
sess.Config.Credentials = stscreds.NewCredentials(sess, n.conf.Sigv4.RoleARN)
}

data := notify.GetTemplateData(ctx, n.tmpl, alert, n.logger)
tmpl := notify.TmplText(n.tmpl, data, &err)
message := tmpl(n.conf.Message)

client := sns.New(sess, &aws.Config{Credentials: credentials})
publishInput := &sns.PublishInput{}

if n.conf.TopicARN != "" {
publishInput.SetTopicArn(n.conf.TopicARN)
messageToSend, isTrunc, err := validateAndTruncateMessage(message)
if err != nil {
return false, err
}
if isTrunc {
n.conf.Attributes["truncated"] = "true"
treid314 marked this conversation as resolved.
Show resolved Hide resolved
}
publishInput.SetMessage(messageToSend)
}
if n.conf.PhoneNumber != "" {
publishInput.SetPhoneNumber(n.conf.PhoneNumber)
// If SMS message is over 1600 chars, SNS will reject the message.
_, isTruncated := notify.Truncate(message, 1600)
if isTruncated {
return false, fmt.Errorf("SMS message exeeds length of 1600 charactors")
treid314 marked this conversation as resolved.
Show resolved Hide resolved
} else {
publishInput.SetMessage(message)
}
}
if n.conf.TargetARN != "" {
publishInput.SetTargetArn(n.conf.TargetARN)
messageToSend, isTrunc, err := validateAndTruncateMessage(message)
if err != nil {
return false, err
}
if isTrunc {
n.conf.Attributes["truncated"] = "true"
}
publishInput.SetMessage(messageToSend)
}

if len(n.conf.Attributes) > 0 {
attributes := map[string]*sns.MessageAttributeValue{}
treid314 marked this conversation as resolved.
Show resolved Hide resolved
for k, v := range n.conf.Attributes {
attributes[k] = &sns.MessageAttributeValue{DataType: aws.String("String"), StringValue: aws.String(v)}
}
publishInput.SetMessageAttributes(attributes)
}

if n.conf.Subject != "" {
publishInput.SetSubject(n.conf.Subject)
}

// Deduplication key is only added if it's a FIFO SNS Topic.
if n.conf.IsFIFOTopic {
key, err := notify.ExtractGroupKey(ctx)
if err != nil {
return false, err
}
publishInput.SetMessageDeduplicationId(key.Hash())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that's part of the original specifications but I'm not sure I fully understand why we set it to the group key's hash. To my understanding (but I may be wrong) is that the 5-minutes deduplication time window done by SNS will conflict with the Alertmanager group_interval if it's set less to 5 minutes because we may loose new alerts added to the group because of the SNS deduplication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I confirmed the group key hash isn't unique enough to handle a message with the same title but different labels send in the 5 minute SNS de-duplication window. I'll ask in the issue, we're also missing a message group id for FIFO queues that is required by AWS.

}

publishOutput, err := client.Publish(publishInput)
if err != nil {
// AWS Response is bad, probably a config issue.
treid314 marked this conversation as resolved.
Show resolved Hide resolved
return false, err
treid314 marked this conversation as resolved.
Show resolved Hide resolved
}

err = n.logger.Log(publishOutput.String())
if err != nil {
return false, err
}

// Response is good and does not need to be retried.
treid314 marked this conversation as resolved.
Show resolved Hide resolved
return false, nil
}

func validateAndTruncateMessage(message string) (string, bool, error) {
if utf8.ValidString(message) {
treid314 marked this conversation as resolved.
Show resolved Hide resolved
// if the message is larger than 256KB we have to truncate.
if len(message) > 256*1024 {
treid314 marked this conversation as resolved.
Show resolved Hide resolved
truncated := make([]byte, 256*1024, 256*1024)
copy(truncated, message)
return string(truncated), true, nil
}
return message, false, nil
}
return "", false, fmt.Errorf("non utf8 encoded message string")
}

// New returns a new SNS notification handler.
func New(c *config.SNSConfig, t *template.Template, l log.Logger, httpOpts ...commoncfg.HTTPClientOption) (*Notifier, error) {
treid314 marked this conversation as resolved.
Show resolved Hide resolved

treid314 marked this conversation as resolved.
Show resolved Hide resolved
client, err := commoncfg.NewClientFromConfig(*c.HTTPConfig, "sns", append(httpOpts, commoncfg.WithHTTP2Disabled())...)
if err != nil {
return nil, err
}

return &Notifier{
conf: c,
tmpl: t,
logger: l,
client: client,
retrier: &notify.Retrier{},
}, nil
}