Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: QoL improvements for benchmarks #1609

Merged
merged 4 commits into from Dec 7, 2022

Conversation

ddyurchenko
Copy link
Collaborator

@ddyurchenko ddyurchenko commented Dec 7, 2022

What does this PR do?

  1. Change threshold for marking a difference in benchmark results as a statistically significant.
  2. Enable S3 bucket uploading, since permissions are already configured.
  3. Change how PR-commenter acts on a new benchmark result - instead of minimizing a PR comment, replace it, like it is done in the most CI/CD tools and in dd-trace-js.

Motivation

Improve benchmark workflows based on previous requests from #apm-benchmarking-platform channel.

Describe how to test/QA your changes

Successful S3 bucket uploading: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-go/-/jobs/201494066

Reviewer's Checklist

  • If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
  • Changed code has unit tests for its functionality.
  • If this interacts with the agent in a new way, a system test has been added.

@ddyurchenko ddyurchenko requested a review from a team as a code owner December 7, 2022 11:02
@ddyurchenko ddyurchenko force-pushed the ddyurchenko/qol-improvements-benchmarks branch from 4c3ad24 to 2780eb4 Compare December 7, 2022 11:54
@pr-commenter
Copy link

pr-commenter bot commented Dec 7, 2022

Benchmarks

Comparing candidate commit 20b7553 in PR branch ddyurchenko/qol-improvements-benchmarks with baseline commit cf23647 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 6 cases.

@ajgajg1134 ajgajg1134 added this to the v1.45.0 milestone Dec 7, 2022
ajgajg1134
ajgajg1134 previously approved these changes Dec 7, 2022
@ddyurchenko ddyurchenko force-pushed the ddyurchenko/qol-improvements-benchmarks branch from 04381fd to 20b7553 Compare December 7, 2022 16:38
@ddyurchenko
Copy link
Collaborator Author

Hey @ajgajg1134 ,

Can you please review & merge this PR?

I changed 2 things:

  1. Now baseline and candidate commits are guaranteed to be in correct order for comparison.
  2. Fixed plots in benchmark-analyzer that were distorted due to long labels.

Thanks!

@ajgajg1134 ajgajg1134 changed the title QoL improvements for benchmarks ci: QoL improvements for benchmarks Dec 7, 2022
@ajgajg1134 ajgajg1134 merged commit 0ab9332 into main Dec 7, 2022
@ajgajg1134 ajgajg1134 deleted the ddyurchenko/qol-improvements-benchmarks branch December 7, 2022 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants