Prometheus metric handler is exposing 500 server error

[geet]

I have instrumented custom prometheus metrics from my application. I have configured prometheus to scrape the target. I have started seeing 500 error in the prometheus and due to that the scrape failed. I tried to manually hit the /metrics endpoint in my application and got the error saying

<metric> [label_values] was collected before with the same name and label values

It works initially but after a few minutes I start seeing the error.

I don't think it should be considered a duplicate metric but if the metric with same label and values are added,then it should increase the count.

go client version: github.com/prometheus/client_golang v1.12.2

[geet]

Here is the code snippet I am using to provide prometheus handler with gofiber.

import (
“github.com/gofiber/adaptor/v2](http://github.com/gofiber/adaptor/v2”
“github.com/prometheus/client_golang/prometheus/promhttp"

)

func MetricsRoutes(api fiber.Router) {
api.Get(“/metrics”, adaptor.HTTPHandler(promhttp.Handler()))
}

[geet]

The problematic metric is a custom GaugeVec registerd by the following code.

func RegisterMetric(metricNames []string) {
	for i, metricName := range metricNames {
		metricGauge := prometheus.NewGaugeVec(prometheus.GaugeOpts{
			Name: metricName,
		}, getLabels(metricName))
		metricsList[i] = metricName
		metrics[i] = metricGauge
		prometheus.MustRegister(metricGauge)
	}
}

getLabels returns the labels supported.

I see the metric being collected fine for the first few scrapes and suddenly I start seeing 500 server error on the scrapes.

The error thrown is like this.

* collected metric "en_devices" { label:<name:"_storage_" value:"3" > label:<name:"ctype" value:"public" > label:<name:"en_destination" value:"iOS " > label:<name:"en_device_type" value:"A" > label:<name:"location" value:"us" > label:<name:"scope" value:"<some_string>" > label:<name:"service_instance" value:"<some_guid>" > label:<name:"service_name" value:"<some_string>" > gauge:<value:1 > } was collected before with the same name and label values

[bwplotka]

Hi, It looks like your instrumentation collected the same metric twice from the all metrics (e.g. gauges you showed) and custom collectors in your application. Looks like you are using the default registry (prometheus.DefaultRegistry).

It's unlikely the presented code is responsible for the problem. The MustRegister checks in duplication in https://github.com/prometheus/client_golang/blob/1f81b3e9130fe21c55b7db82ecb2d61477358d61/prometheus/registry_test.go#L750[metric definition]. Try running RegisteMetric twice with the same metric names, and you will see it will panic.

When RegisterMetric is invoked once I don't see it returning duplicated collections, since MustRegister ensures that. It must be something else inside your application that transparently registers custom collector that does not define metric properly and returns metrics that perhaps collide with gauges you create in RegisterMetric.

My recommendation is to NEVER use default registry as it causes the problems like here, where we think we know about all the collectors registers, but in fact any package might we import might implicitly register something funky. I suggest creating your own registry reg := prometheus.NewRegistry() (implements prometheus.Gatherer used by handler and prometheus.Registerer for metrics) and pass it to all users and handlers:

import (
“github.com/gofiber/adaptor/v2"
“github.com/prometheus/client_golang/prometheus/promhttp"

)

func MetricsRoutes(api fiber.Router, reg prometheus.Gatherer) {
api.Get(“/metrics”, adaptor.HTTPHandler(promhttp.HandlerFor(reg, nil)))
}

func RegisterMetric(metricNames []string, reg prometheus.Gatherer) {
	for i, metricName := range metricNames {
		metricGauge := promauto.With(reg).NewGaugeVec(prometheus.GaugeOpts{
			Name: metricName,
		}, getLabels(metricName))
		metricsList[i] = metricName
		metrics[i] = metricGauge
	}
}

With this you will know explicitly that it is the only collector handler is using. Otherwise you have to track exactly all collectors and check who is creating en_devices metric.

TODO(#1101): Document this path

[geet]

In this code snippet

func RegisterMetric(metricNames []string, reg prometheus.Gatherer) {
	for i, metricName := range metricNames {
		metricGauge := promauto.With(reg).NewGaugeVec(prometheus.GaugeOpts{
			Name: metricName,
		}, getLabels(metricName))
		metricsList[i] = metricName
		metrics[i] = metricGauge
	}
}

I guess instead of passing a Gatherer, it needs to accept a prometheus.Registerer type

[geet]

Hi, I made the changes as suggested and still seeing errors from the prometheus client. There are multiple occurrence of such errors.

was collected before with the same name and label values

error gathering metrics: 42 error(s) occurred

[mihail641]

@geet, you are using Fiber, in Fiber http parameters, body unsafe and mutable. Set Immutable: true in fiber.Config.
Or you can copy unsafe string value in immutable string fmt.Sprinf("%s", myVal)