Consistent Benchmark Result

[shayff]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the issue has not already been raised

Issue

Hey,
I'm trying to do some performance improvement research and therefore I tried to run the benchmark for Fastify a few times and I got a bit different result.

For example:

• First run I got total of 1,066,024 requests
• Second run I got total of 1,344,495 requests

A difference of approximately 20% without any changes.
How can I get more consistent result?

[Eomm]

How can I get more consistent result?

You need a controlled environment.
If the test PC starts something (a calendar notification or a new connection to something) impacts a running bench.

For this reason the benchmarks are run in the same GH instance here

[arthurfiorette]

Hey @Eomm, are they being run in a dedicated CI environment? I tried running them in a Free Github Actions Runner and it still got me different results every time.

https://github.com/kitajs/benchmarks/actions/workflows/benchmarks.yml

[mcollina]

We are not running them on dedicated HW, just standard GH actions. I run them on my dedicated machine when I pull them for Fastify main repo.