Features Benchmark Snapshot

[workgroupengineering]

This is a preliminary version.

The purpose

The purpose of this PR is to make it possible to compare benchmarks between two versions or run when nuget packages cannot be used.

How works

Use the ability to create custom toolchains to inject ExecuteResults previously saved into a store. To associate methods between two executions, hash of BenchmarkCase.Descriptor. At this time only the json store is implemented.

Pro

• it is immune to breaking changes
• not breaking changes (add only InternalsVisibleTo to BenchmarkDotNet bcl)
• Plugable store

Cons

• none

Known Issues

• cannot retrieve IterationCount, LaunchCount, WarmupCount of a Snapshot.
• I am unable to obtain the path of artifacts in the Snapshot restore phase

Example of output

BenchmarkDotNet=v0.13.1.20220414-develop, OS=Windows 10 (10.0.19044.1586/21H2/November2021Update)
Intel Core i7-7700K CPU 4.20GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.300-preview.22154.4
  [Host]   : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT DEBUG  [AttachedDebugger]
  Baseline : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT
  ShortRun : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT

Method	Job	Toolchain	IterationCount	LaunchCount	WarmupCount	Mean	Error	StdDev	Ratio	RatioSD
Sleep	Baseline	JsonShanpshotStore	Default	Default	Default	15.63 ms	1.365 ms	0.075 ms	1.00	0.00
Sleep	ShortRun	Default	3	1	3	16.67 ms	2.897 ms	0.159 ms	1.07	0.02
Thread.Sleep(15)	Baseline	JsonShanpshotStore	Default	Default	Default	16.82 ms	4.494 ms	0.246 ms	1.00	0.00
Thread.Sleep(15)	ShortRun	Default	3	1	3	17.53 ms	19.891 ms	1.090 ms	1.04	0.08

[workgroupengineering]

@adamsitnik @AndreyAkinshin do you have any suggestions to improve the quality ad fix issue of this PR?
It was originally intended as a separate repository, but I need to access internal classes.

[YegorStepanov]

compare benchmarks between two versions

A cmd tool is better suited for this, which merges the results based on the BDN artifact files.

when nuget packages cannot be used

Can you clarify what you mean?

[timcassell]

Looks like you misspelled Snapshot in several places.

[workgroupengineering]

Looks like you misspelled Snapshot in several places.
@timcassell Thank you for report

@YegorStepanov

compare benchmarks between two versions

A cmd tool is better suited for this, which merges the results based on the BDN artifact files.

Why? With the approach I suggested, you can have the comparison in all existing exporters using the engine of
BenchmarkDotNet for comparison.

when nuget packages cannot be used

Can you clarify what you mean?

In my spare time I am developing a benchmark comparator for avalonia. But maintainers don't want to publish Avalonia.Base.UnitTests as a nuget.

[workgroupengineering]

What do you think if you base the implementation of this functionality above the PR #2178?