Prepare v2.52.0-rc.0 release

[ArthurSens]

There were a few entries which I was not sure what to announce. Please let me know if you have better suggestions :)

[bwplotka]

Nice, a couple of suggestions, thanks!

[bwplotka]

Nice! We spent some time syncing with @ArthurSens resulting in those notes

[bwplotka]

We could make it a feature or join with stats bugfix

[ArthurSens]

Thinking again about this, one is counting in WAL checkpoint and the other is counting in query stats... I'm not sure if it makes sense to join them in a single changelog entry

[ArthurSens]

/prombench v2.51.2

[prombot]

@ArthurSens is not a org member nor a collaborator and cannot execute benchmarks.

[Nexucis]

/prombench v2.51.2

[prombot]

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-13969 and v2.51.2

After successful deployment, the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart v2.51.2

[ArthurSens]

Small decrease in memory allocations

I can see a very small increase in CPU, not sure how much is relevant enough though

I also see small increases in all query steps, but they are nanoseconds, or very small miliseconds

[prombot]

Benchmark tests are running for 3 days! If this is intended ignore this message otherwise you can cancel it by commenting: /prombench cancel

[ArthurSens]

/prombench cancel

[prombot]

Benchmark cancel is in progress.

[ArthurSens]

Initially, I thought the latency in different query steps would be insignificant, but when looking at overall query latency I can see a significant impact on the 0.99 quantile.

Query used:

sum(prometheus_engine_query_duration_seconds{namespace="prombench-${prNumber}"}) by (prometheus, quantile)

[bwplotka]

Retro link: http://prombench.prometheus.io/grafana/d/7gmLoNDmz/prombench?orgId=1&from=1713875299363&to=1714140211760&var-RuleGroup=All&var-prNumber=13969

[bwplotka]

Also noticed uneven traffic to services, explaining latency being higher for Prom from this PR for the period where it has more request to serve than the old one. Unfortunately it still happens but we improved this a lot already (uneven query load distribution):

[bwplotka]

Unfortunately we see even 0.9 percentile difference for moments the load was even. It seems indeed we have a small regression in query tail latency in this release, but perhaps something to tolerate. Many optimizations had happened and could impact this.

[bwplotka]

For 0.99 it was sometimes extreme difference - correlating with CPU does not help (maybe a bit more CPU used, weird)

[bwplotka]

I think that one was affected by compaction (unlucky)

[bwplotka]

Bit sketchy, there is a slight different in tail latency for queries, but perhaps negligible.

Either we stop release and try to bisect what optimization could cause this OR we proceed and optimize later when (if) users will be actually affected by this, ideally in RC phase. Happy with either option.

[ArthurSens]

Hmmm, I didn't notice the compaction happening at the same time, but the compaction happens for both versions and only 1 spikes. The uneven query distribution sounds like a good explanation tho

I feel like this release candidate was delayed enough already, let's try publishing it and see if we get more feedback about slow queries.