custom metric state - could we keep it on disk?

[RichardWright]

We rely alot on metrics and the share of memory that prometheus uses is growing. Has it ever been suggested that counters/gauges could write to disk and retrieve that state for scraping?

Thanks for the great lib btw. We've recently moved off open telemetry and really appreciate the flexibility of the prom-client api, setting gauges is a great feature.

[shappir]

Saving to disk will make updating entires much more expensive - I don't think it's a viable approach in this case.

I would recommend looking at ways to reduce the cardinality of your data, especially histograms: reduce the number of labels, label value permutations, buckets.

There are ways in which the library itself could be improved, e.g. currently each histogram entry contains a dictionary of its label values, which is essentially a duplication of the data used as entry index key. A more memory efficient data structure could be used instead.

[RichardWright]

We gave up on histograms a while ago.

Memory efficieny would be appreciated. We have quite high cardinality as we run stateful, long lived, socket based applications and use the metrics as a way of shipping that state. Ideally we wouldn't, but an alternative pattern isn't obvious.