High Memory Usage after upgrade v2.34 -> v2.43

[mramesh047]

What did you do?

Prometheus version upgrade from 2.34 to 2.43

What did you expect to see?

memory footprint reduction

What did you see instead? Under which circumstances?

We see higher memory usage

sum(container_memory_working_set_bytes{pod="prometheus*",container="prometheus"}) by (container)/sum(pod="prometheus*",container="prometheus"}) by (container) * 100

There is No increase in go_memstats_heap_inuse_bytes
There is a surge in container_memory_rss correlating to above spike

System information

CentOS Linux release 7.9.

Prometheus version

Build Information
Version	2.43.0
Revision	non-git
Branch	non-git
BuildDate	20230709-02:02:42
GoVersion	go1.20.5

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[roidelapluie]

Unfortunately we can not support non-official builds. In this particular case, it is unclear if your build has been done with the same go tags as the official release.

[mramesh047]

Thanks @roidelapluie for pointing it out. Will run the build with stringlabels tag