Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-activate automated benchmark tests #2114

Open
mrickard opened this issue Apr 2, 2024 · 1 comment
Open

Re-activate automated benchmark tests #2114

mrickard opened this issue Apr 2, 2024 · 1 comment

Comments

@mrickard
Copy link
Member

mrickard commented Apr 2, 2024

In the past, agent teams measured performance using the OATS system, which required a lot of hands-on attention to operate. Our agent repo contains benchmark tests that are runnable locally, and which could be easily incorporated in a regular workflow.

npm run bench runs the existing benchmark test suite. Currently some fail due to changes in the agent or changes to agent dependencies. Results are sent to stdout.

Anticipated related tickets:

  • Spike(s) to determine which tests are being run: which pass, which fail, which to keep, which failing ones to fix, which to remove, and which new tests to add.
  • Alter benchmark tests to ship results to New Relic, instead of stdout.
  • Create a dashboard for presenting bench results correlated to agent version as well as Node runtime
  • Create workflow for running benchmarks on PRs and nightly.

Acceptance Criteria

  • Failing benchmark tests should be fixed or removed
  • There should be a dashboard in NR we can view to graph performance over time, tagged with agent version and Node runtime.
  • There should be GHA workflows that run a matrix of currently supported Node runtimes on the current agent version.
  • Running benchmark tests should result in telemetry sent to NR, and that telemetry should be presented in that dashboard.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Node.js Engineering Board
  
Triage Needed: Unprioritized Features
Development

No branches or pull requests

2 participants