Prometheus: Restore FromAlert header #55255
Conversation
|
Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/33660 |
|
Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/33971 |
kylebrandt
changed the title
|
@aocenas Are you able to check this out since it reverts a PR of yours? |
| addHeaders := make(map[string]string) | ||
|
|
||
| if req.Headers["FromAlert"] == "true" { | ||
| addHeaders["FromAlert"] = "true" |
There was a problem hiding this comment.
Is this only for the FromAlert info? Out of curiosity why is it needed to send to prometheus? Could it be moved to infra code so that it is always sent downstream?
There was a problem hiding this comment.
Only for the header. Since it was being sent, people have used it to load balance on. So they point their datasource at their LB and use the header to route differently.
Not sure I follow with moving to the infra code? This doesn't seem like the right way to fix it to me, but trying to undo the experienced regression for the user.
There was a problem hiding this comment.
Yep got it, this works so no issue in merging.
Not sure I follow with moving to the infra code?
We are already handling headers in grafana backend in general by providing a HTTP client to datasource, so it does not have to be handled custom, which is why this code was removed. We may somehow move this to the HTTP client that is provided so that FromAlert header would be passed upstream for all datasources automatically if it is valuable in general.
What this PR does / why we need it:
Restores the FromAlert header to prometheus that comes for Grafana Managed Alert queries.
Which issue(s) this PR fixes:
Fixes #54403
Special notes for your reviewer:
I think was made a feature by accident, but people are using Load balance / route queries to different higher SLA servers for alerting.
So this doesn't fix that it isn't thought out as feature (e.g. document, support across datasource, less fragile internal code then a header). But it does provide a fix of restoring the header.
This reverts #51518 , but with the change to only pass through the FromAlert header.