opsgenie_config using api_key_file not working

[zoezhangmattr]

What did you do?
using vault injector to inject the api key - has issue
the /vault/secrets/opsgenie_api_key , the content is the apikey
file owner is nobody, same as alertmanager user/group,. its mode is 644 or 777, tried both

same alert can be routed to slack, but cant be opsgenie

using plain text api key value - works
What did you expect to see?
thought it should work, but so far no luck, need some guidance pls
What did you see instead? Under which circumstances?
ts=2024-03-13T02:42:59.388Z caller=notify.go:848 level=warn component=dispatcher receiver=opsgenie integration=opsgenie[0] aggrGroup="{}/{severity=~"^(?:critical|error)$"}:{}" msg="Notify attempt failed, will retry later" attempts=1 err="Post "https://api.opsgenie.com/v2/alerts\": net/http: invalid header field value for "Authorization""
Environment

• System information:

  insert output of uname -srm here

• Alertmanager version:

  insert output of alertmanager --version here (repeat for each alertmanager
  version in your cluster, if relevant to the issue)
  0.26.0 and 0.27.0

• Prometheus version:

  insert output of prometheus --version here (repeat for each prometheus
  version in your cluster, if relevant to the issue)
  2.47.0

• Alertmanager configuration file:

global: {}
receivers:
- name: opsgenie
  opsgenie_configs:
  - api_key_file: /vault/secrets/opsgenie_api_key
    message: "{{ range .Alerts }} \n{{ .Annotations.summary }}\n{{ end }}"
    priority: '{{ if .CommonAnnotations.priority }}{{ .CommonAnnotations.priority
      }}{{ else }}P3{{ end }}'
    responders:
    - name: devops
      type: team
route:
  group_interval: 5m
  group_wait: 10s
  receiver: alerts-slack
  repeat_interval: 3h
  routes:
  - continue: true
    match_re:
      severity: critical|error
    receiver: opsgenie

• Logs:

ts=2024-03-13T02:42:59.388Z caller=notify.go:848 level=warn component=dispatcher receiver=opsgenie integration=opsgenie[0] aggrGroup="{}/{severity=~\"^(?:critical|error)$\"}:{}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://api.opsgenie.com/v2/alerts\": net/http: invalid header field value for \"Authorization\""

[TheMeier]

This looks like something is wrong with the api key and not alertmanager. Have you verified, eg in a test pod, that /vault/secrets/opsgenie_api_key really contains the correct key?

[zoezhangmattr]

This looks like something is wrong with the api key and not alertmanager. Have you verified, eg in a test pod, that /vault/secrets/opsgenie_api_key really contains the correct key?

thanks for reply,
yes, the file has the correct key id, funny thing is using the same way to do opsgenie heatbeat, using same key, works for deadman switch

- name: prometheus-deadman-switch
  webhook_configs:
  - url: https://api.opsgenie.com/v2/heartbeats/xxxxxx/ping
    send_resolved: false
    http_config:
      basic_auth:
        username: ':'
        password_file: /vault/secrets/opsgenie_api_key

[TheMeier]

One is an opsgenie_configs the other is a http_config. You are using /vault/secrets/opsgenie_api_key as a password in the latter indicating to me that it contains a paassword and not an API key.

[TheMeier]

@zoezhangmattr any feedback?

[zoezhangmattr]

One is an opsgenie_configs the other is a http_config. You are using /vault/secrets/opsgenie_api_key as a password in the latter indicating to me that it contains a paassword and not an API key.

no, as i mentioned before, the api key is working if using k8s secret , same api key, the password is correct, it should be opsgenie api key in this case

[grobinson-grafana]

Hi @zoezhangmattr! Does the file exist and contain the secret at the time the Alertmanager is started? It sounds like there might be a race condition between the Alertmanager starting and vault-injector writing the file.