Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unnecessary allocations in Prometheus scraping endpoint #30085

Closed

Conversation

stokpop
Copy link
Contributor

@stokpop stokpop commented Mar 7, 2022

Reduce the number of times capacity growth is needed inside the StringWriter. A typical default SpringBoot Prometheus page has more than 11k characters. Best performance results when no capacity growth is needed at all, so base it on previous metrics page size plus some room for possible extra metric info.

Background

Profiling our SpringBoot application showed time spend in the StringWriter of PrometheusScrapeEndpoint, and below that some time spend in Arrays.copy of the wrapped StringBuilder.ensureCapacity. Default size is 16, so there will be multiple new array allocations and array copies during the filling of the StringWriter.

This is below TextFormat.writeOpenMetrics100:
image

A quick gain might be to pre size the StringWriter in PrometheusScrapeEndpoint.

A JMH benchmark on local Mac M1 using StringWriter with different initial capacity show no substantial gain in ops/s. Effect might be different/better in virtualized container environment with less cpu power? The JMH benchmark does show significant memory allocation improvement if StringWriter is created with initial actual size + some: can be around 3 times less.

Bytes allocated per op, with different initial StringWriter sizes:
image

Bytes allocated with StringWriter page size + 2:
image

First tried with fixed 14 * 1024 size, but that turns out to make it possibly worse when metrics page is a bit bigger than that and dynamic increase of backing array is 'too large' (e.g. see last column in first table). So then introduced a previousMetricsScrapeSize to hold size of last scrape and make new StringWriter that size plus some more for possible new data.

Not sure about actual improvement, but gut feeling is: less memory allocation, less objects (arrays) created to be garbage collected, less copying of arrays, and a component that is running in many places on earth will save some.

Final note: A default Prometheus metrics page is about 11k-12k characters. Adding additional metrics, e.g. via micrometer, will increase the size. We found that we used MicrometerHttpClientInterceptor that would add one line of metric info for each unique url, so that made 800k+ pages and had this issue magnified in profiling.

Reduce the number of times capacity growth is needed inside the StringWriter. A typical default SpringBoot Prometheus page has more than 11k characters. Best performance results when no capacity growth is needed at all, so base it on previous metrics page size plus some room for possible extra metric info.
@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Mar 7, 2022
@mhalbritter
Copy link
Contributor

Hey @stokpop, thanks for the PR!

I think the idea to remember the last scrape size is very nice. But i would not set the initial size (the one before the 1st scrape) to a fixed 12kb, instead we should use the defaults of the StringWriter in that case. Can you please update the PR? Thanks a lot!

@mhalbritter mhalbritter added this to the 2.7.x milestone Mar 8, 2022
@mhalbritter mhalbritter added status: waiting-for-feedback We need additional information before we can continue and removed status: waiting-for-triage An issue we've not yet triaged labels Mar 8, 2022
…itial 12k but the StringWriter default size.
@stokpop
Copy link
Contributor Author

stokpop commented Mar 8, 2022

@mhalbritter good idea, so that would be 16, the default size of StringWriter, I have made the update.

@spring-projects-issues spring-projects-issues added status: feedback-provided Feedback has been provided and removed status: waiting-for-feedback We need additional information before we can continue labels Mar 8, 2022
mhalbritter pushed a commit that referenced this pull request Mar 9, 2022
Reduce the number of times capacity growth is needed inside the StringWriter.
A typical default SpringBoot Prometheus page has more than 11k characters.
Best performance results when no capacity growth is needed at all, so base
it on previous metrics page size plus some room for possible extra metric info.

See gh-30085
mhalbritter added a commit that referenced this pull request Mar 9, 2022
…nt"\n\nSee gh-30085 from

* 2.6.x-prometheus-performance:
  Polish "Tweak performance for Prometheus scraping endpoint"\n\nSee gh-30085
  Tweak performance for Prometheus scraping endpoint\n\nCloses gh-"Tweak performance for Prometheus scraping endpoint"\n\nSee gh-30085
@mhalbritter mhalbritter modified the milestones: 2.7.x, 2.7.0-M3 Mar 9, 2022
@mhalbritter
Copy link
Contributor

Merged, thanks a lot for the contribution!

@mhalbritter mhalbritter closed this Mar 9, 2022
@mhalbritter mhalbritter added type: enhancement A general enhancement and removed status: feedback-provided Feedback has been provided labels Mar 9, 2022
@mhalbritter mhalbritter self-assigned this Mar 9, 2022
@wilkinsona wilkinsona changed the title Performance tweak for Prometheus scraping endpoint. Avoid unnecessary allocations in Prometheus scraping endpoint Mar 9, 2022
@wilkinsona wilkinsona changed the title Avoid unnecessary allocations in Prometheus scraping endpoint Unnecessary allocations in Prometheus scraping endpoint Mar 9, 2022
@wilkinsona
Copy link
Member

@mhalbritter and I discussed this a bit and we've decided that the unnecessary allocations are a performance bug.

@wilkinsona wilkinsona added type: bug A general bug and removed type: enhancement A general enhancement labels Mar 9, 2022
@wilkinsona wilkinsona modified the milestones: 2.7.0-M3, 2.5.x Mar 9, 2022
@wilkinsona wilkinsona reopened this Mar 9, 2022
mhalbritter pushed a commit that referenced this pull request Mar 9, 2022
Reduce the number of times capacity growth is needed inside the StringWriter.
A typical default SpringBoot Prometheus page has more than 11k characters.
Best performance results when no capacity growth is needed at all, so base
it on previous metrics page size plus some room for possible extra metric info.

See gh-30085
@mhalbritter mhalbritter modified the milestones: 2.5.x, 2.5.11 Mar 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants