(This file is generated by py . lint)
A benchfile lets us vary three different things: coreclrs, configs, and benchmarks. The test runner uses all combinations of coreclrs ⨯ configs ⨯ benchmarks. Each is a map with keys being arbitrary names.
Here's an example benchfile:
coreclrs:
clr_a:
core_root: coreclr
commit_hash: 930abba4060fb528db2bb9835a1bc5a6e684bfec
clr_b:
core_root: coreclr2
commit_hash: ed52a006c01a582d4d34add40c318d6f324b99ba
options:
collect: gc
default_iteration_count: 3
common_config:
complus_gcserver: true
complus_gcconcurrent: false
configs:
smaller:
complus_gcgen0size: 16777216
bigger:
complus_gcgen0size: 33554432
benchmarks:
nosurvive:
executable: GCPerfSim
arguments:
tc: 8
tagb: 500
tlgb: 1
lohar: 0
sohsi: 50
lohsi: 0
sohpi: 0
lohpi: 0
sohfi: 0
lohfi: 0
allocType: reference
testKind: time
comment: str | None
(ignored)
vary: "machine" | "coreclr" | "config" | "benchmark" | None
Preferred property to vary when using py . diff
configs_vary_by: [ConfigsVaryBy](#ConfigsVaryBy) | None
This is mostly set just for information.
When there are many configs, this describes the one property that is changing.
coreclrs: Mapping[str, [CoreclrSpecifier](#CoreclrSpecifier)]
Mapping from an (arbitrary) coreclr name to its specifier.
paths: Mapping[str, Path] | None
Mapping of shorthand names for paths.
If the 'executable' field of a Benchmark is a key in this mapping,
it will be replaced with the corresponding value.
options: [BenchOptions](#BenchOptions)
Additional options that apply to every test.
common_config: [Config](#Config) | None
Config properties common to all configs.
Properties set here should not overlap with anything in 'configs'.
If omitted, the common config is empty.
configs: Mapping[str, [Config](#Config)] | None
Mapping from an (arbitrary) config name to the config.
Unlike coreclrs and benchmarks, this section is optional.
If omitted, common_config will be used.
benchmarks: Mapping[str, [Benchmark](#Benchmark)]
Mapping from an (arbitrary) benchmark name to the benchmark.
scores: Mapping[str, Mapping[str, [ScoreElement](#ScoreElement)]] | None
Mapping from an (arbitrary) score name to its specifier.
collect: "none" | "gc" | "verbose" | "cpu_samples" | "cswitch" | None
Kind of events to collect.
Defaults to 'gc'.
none: Do not collect any trace events. gc: Collect normal GC events. verbose: Collect verbose GC events, which includes join events. cpu_samples: Collect all of the above, and CPU samples. cswitch: Collect all of the above, and CSwitch events on Windows. No effect on Linux.
default_iteration_count: int | None
Number of times to run the same test combination.
"Defaults to 1.
"Without this you will not get stdev.
default_min_seconds: float | None
If a test is over in less than this many seconds, the test runner will throw an exception.
This ensures that tests have sufficient data for performance analysis.
May be overridden by Benchmark#min_seconds
default_max_seconds: float | None
The test runner will stop a process that goes longer than this and fail.
May be overridden by Benchmark#max_seconds.
max_trace_size_gb: float | None
If the trace exceeds this size, old events will be discarded.
Defaults to 1.
dotnet_path: Path | None
Custom 'dotnet' path to use when building.
Does not affect running.
dotnet_trace_path: Path | None
Set this in case dotnet-trace is not in PATH.
Useful for tests that must be run as super user.
dotnet_trace_buffersize_mb: int | None
Value to pass to --buffersize argument of dotnet-trace, in MB.
Default is dotnet-trace's default, currently 256MB.
always_use_dotnet_trace: bool | None
Always use dotnet-trace to collect traces, even on Windows where PerfView is the default.
Has no effect on non-Windows.
Has no effect if collect is none.
executable: str | None
Path or key into 'paths' from the BenchFile.
Defaults to "GCPerfSim".
arguments: str | [GCPerfSimArgs](#GCPerfSimArgs) | None
Command line arguments to pass to the executable.
For GCPerfSim, you can also specify GCPerfSimArgs, and it will be converted to a string for you.
iteration_count: int | None
Defaults to options.default_iteration_count
min_seconds: float | None
Defaults to options.default_min_seconds
max_seconds: float | None
Defaults to options.default_max_seconds
only_configs: Sequence[str] | None
Only run the test against configs with one of the specified names.
Allows to set environment variables, and container and memory load options. WARN: Normally complus environment variables are specified in hexadecimal on the command line. But when specifying them in a yaml file, use decimal.
complus_gcserver: bool | None
Set to true to use server GC.
complus_gcconcurrent: bool | None
Set to true to allow background GCs.
complus_gcgen0size: int | None
gen0size in bytes. (decimal)
complus_gcgen0maxbudget: int | None
Max gen0 budget in bytes. (decimal)
complus_gcheapaffinitizeranges: str | None
On non-Windows, this should look like: 1,3,5,7-9,12
On Windows, this should include group numbers, like: 0:1,0:3,0:5,1:7-9,1:12
complus_gcheapcount: int | None
Number of heaps. (decimal)
Only has effect when complus_gcserver is set.
complus_gcheaphardlimit: int | None
Hard limit on heap size, in bytes. (decimal)
complus_gclargepages: bool | None
Set to true to enable large pages.
complus_gcnoaffinitize: bool | None
Set to true to prevent affinitizing GC threads to cpu cores.
complus_gccpugroup: bool | None
Set to true to enable CPU groups.
complus_gcnumaaware: bool | None
Set to false to disable NUMA-awareness in GC
complus_thread_useallcpugroups: bool | None
Set to true to automatically distribute threads across CPU Groups
complus_threadpool_forcemaxworkerthreads: int | None
Overrides the MaxThreads setting for the ThreadPool worker pool
complus_tieredcompilation: bool | None
Set to true to enable tiered compilation
complus_bgcfltuningenabled: bool | None
Set to true to enable dotnet/coreclr#26695
complus_bgcmemgoal: int | None
See comment on dotnet/coreclr#26695
complus_bgcmemgoalslack: int | None
See comment on dotnet/coreclr#26695
complus_gcconcurrentfinalization: bool | None
Enable concurrent finalization (not available in normal coreclr builds)
container: [TestConfigContainer](#TestConfigContainer) | None
Set to run the test in a container.
A container is a job object on Windows, or cgroups / docker container on non-Windows.
affinitize: bool | None
If true, this will be run in a job object affinitized to a single core.
Only works on Windows.
See run_in_job.c's --affinitize option.
memory_load: [MemoryLoadOptions](#MemoryLoadOptions) | None
If set, the test runner will launch a second process that ensures this percentage of the system's memory is consumed.
coreclr_specific: Mapping[str, [ConfigOptions](#ConfigOptions)] | None
Maps coreclr name to config options for only that coreclr.
If present, should have an entry for every coreclr.
complus_gcserver: bool | None
Set to true to use server GC.
complus_gcconcurrent: bool | None
Set to true to allow background GCs.
complus_gcgen0size: int | None
gen0size in bytes. (decimal)
complus_gcgen0maxbudget: int | None
Max gen0 budget in bytes. (decimal)
complus_gcheapaffinitizeranges: str | None
On non-Windows, this should look like: 1,3,5,7-9,12
On Windows, this should include group numbers, like: 0:1,0:3,0:5,1:7-9,1:12
complus_gcheapcount: int | None
Number of heaps. (decimal)
Only has effect when complus_gcserver is set.
complus_gcheaphardlimit: int | None
Hard limit on heap size, in bytes. (decimal)
complus_gclargepages: bool | None
Set to true to enable large pages.
complus_gcnoaffinitize: bool | None
Set to true to prevent affinitizing GC threads to cpu cores.
complus_gccpugroup: bool | None
Set to true to enable CPU groups.
complus_gcnumaaware: bool | None
Set to false to disable NUMA-awareness in GC
complus_thread_useallcpugroups: bool | None
Set to true to automatically distribute threads across CPU Groups
complus_threadpool_forcemaxworkerthreads: int | None
Overrides the MaxThreads setting for the ThreadPool worker pool
complus_tieredcompilation: bool | None
Set to true to enable tiered compilation
complus_bgcfltuningenabled: bool | None
Set to true to enable dotnet/coreclr#26695
complus_bgcmemgoal: int | None
See comment on dotnet/coreclr#26695
complus_bgcmemgoalslack: int | None
See comment on dotnet/coreclr#26695
complus_gcconcurrentfinalization: bool | None
Enable concurrent finalization (not available in normal coreclr builds)
container: [TestConfigContainer](#TestConfigContainer) | None
Set to run the test in a container.
A container is a job object on Windows, or cgroups / docker container on non-Windows.
affinitize: bool | None
If true, this will be run in a job object affinitized to a single core.
Only works on Windows.
See run_in_job.c's --affinitize option.
memory_load: [MemoryLoadOptions](#MemoryLoadOptions) | None
If set, the test runner will launch a second process that ensures this percentage of the system's memory is consumed.
name: str
Name of the property we are testing different values for.
default_values: Mapping[str, float] | None
Value that coreclr would use without an explicit config.
Key is a coreclr name.
self_contained: bool | None
If this is set, benchmark executables should be self-contained.
Then core_root and repo_path should not be set.
core_root: Path | None
Path to a Core_Root directory built from coreclr.
(Does not need to still be located inside a coreclr repository.)
repo_path: Path | None
Instead of specifying Core_Root, you could specify the path to the repository.
Core_Root will be the default location of a release build.
When running on a remote machine, the coreclr path must already exist on that machine. It should be at the same path in every machine being tested. Or, it may be a UNC path.
exact_path: Path | None
Path to the corerun executable.
Generally prefer 'core_root' to this.
commit_hash: str | None
Optional; git commit hash that Core_Root was built from.
On Windows, if SigCheck was installed,
the test runner will verify that CoreRun.exe was tagged with this hash.
On non-Windows this is just for information.
architecture: "amd64" | "x86" | "arm64" | "arm32" | None
On Windows, if SigCheck was installed, the test runner will verify that
CoreRun.exe has the correct bitness corresponding to this architecture.
On non-Windows this is just for information.
Represents the arguments to GCPerfSim. Read the GCPerfSim source for documentation.
tc: int
tagb: float
tlgb: float
totalMins: float | None
lohar: int
sohsi: int
lohsi: int
sohpi: int
lohpi: int
sohfi: int
lohfi: int
allocType: "simple" | "reference"
testKind: "time" | "highSurvival"
percent: float
The memory load process will allocate memory until the system's memory load is this high.
no_readjust: bool | None
If true, the memory load process will never allocate or free any more memory after it's started.
If false, it will allocate or free in order to keep the system's memory at percent.
weight: float
When diffing scores for two traces, this Will be multiplied by the percent difference in a metric.
For example, if config A had a metric value of 2 and config B had a metric value of 3,
and weight is 2, the value of the score is 50% * 2 = 100%.
Weights may be negative to indicate that being below par is preferable.
When considering a single trace alone, the percent difference can come from 'par'. If 'par' is not set, a geometric mean will be used instead.
par: float | None
Expected normal value for this metric.
Options for running the test in a container. A container is a cgroup, or a job object on windows. Docker containers are not yet implemented.
memory_mb: float | None
Size of the container in mebibytes.
cpu_rate_hard_cap: float | None
This allows fractional amounts, e.g. 2.5.
image_name: str | None
Not yet implemented. This would be the name of a docker container.