Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up perf test pipeline and monitoring #790

Open
1 of 2 tasks
Charles-Gagnon opened this issue Apr 5, 2023 · 5 comments
Open
1 of 2 tasks

Set up perf test pipeline and monitoring #790

Charles-Gagnon opened this issue Apr 5, 2023 · 5 comments

Comments

@Charles-Gagnon
Copy link
Contributor

Charles-Gagnon commented Apr 5, 2023

  • Work was started for perf tests such that we have them and can run them, but right now it's a manual effort and not something that we actively run or monitor. We should set it up so it runs on a set schedule and keep an eye out for issues coming from it.
  • Set up monitoring/compare perf numbers to a baseline

https://github.com/Azure/azure-functions-sql-extension/blob/main/builds/azure-pipelines/performance.yml

@MaddyDev
Copy link
Contributor

MaddyDev commented May 1, 2023

Closing this as addressed by #799

@MaddyDev MaddyDev closed this as completed May 1, 2023
@lucyzhang929
Copy link
Contributor

Looks like the perf tests were broken by my integration test refactor - will address that now.

@Charles-Gagnon
Copy link
Contributor Author

@MaddyDev @lucyzhang929 We had also discussed how we intend to use this pipeline, it's not very helpful right now as it is since we don't actually measure anything or compare against any sort of baseline. So we should either re-open this issue for tracking that additional work, or make a separate issue to follow up with getting that done.

@lucyzhang929 lucyzhang929 changed the title Set up perf test pipeline Set up perf test pipeline and monitoring May 1, 2023
@lucyzhang929
Copy link
Contributor

Modified the issue description to track perf monitoring.

@lucyzhang929 lucyzhang929 reopened this May 1, 2023
@MaddyDev MaddyDev self-assigned this Jul 5, 2023
@MaddyDev MaddyDev self-assigned this Jul 19, 2023
@MaddyDev
Copy link
Contributor

MaddyDev commented Oct 6, 2023

Monitoring perf numbers to a baseline plan:

  • For a crawl, we decided to manually add the current runtime numbers as baseline in a json file and as we make improvements, checking manually that we didn't deviate from the baseline numbers.
  • Plugin our telemetry and have a way to get these numbers and have a perf dashboard for us to monitor.
  • For run, have an automatic way to compare the runs against the baseline and fail if they deviate beyond a threshold.

Will be moving this to Post GA after the crawls is completed. (#941)

@MaddyDev MaddyDev modified the milestones: Trigger GA, Post GA Oct 6, 2023
@MaddyDev MaddyDev added the P2 label Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants