When you run processes with bin/cli
, there are two
ways to configure command options:
- on the command line, using the appropriate flag.
- in a configuration file.
To view documentation on the command line,
you can run bin/cli
commands with the --help
option.
For example, to view a summary of the commands available, run:
$ bin/cli --help
To view the documentation for the command-line options
available for the bin/cli run
command, run:
$ bin/cli run --help
Some bin/cli
commands require you to select data for CVEs.
There are three kinds of CVE selector you can use:
- An explicit CVE identifier. For example,
CVE-2018-3743
. - Patterns that select a group of CVEs. For example,
year:2020
,CWE-88
,
andmitre-cwe-top:25:2020
. - A wildcard operator
'*'
that selects all available benchmark CVEs.
To see a full list of the available selectors, run:
$ bin/cli list --help
For commands that require you to select CVEs, you can use any combination of
selectors, and the command will use the union of the selected CVEs.
For example, CWE-88 CWE-89 CVE-123-456
uses the single CVE CVE-123-456
and all CVEs with either CWE-88
or CWE-89
.
For advanced scripting, the selectors can be provided as a line-separated
input stream in stdin
, rather than on the command line. For example:
$ echo CWE-88 > my-cves.txt
$ echo CWE-89 > my-cves.txt
$ echo CVE-123-456 > my-cves.txt
$ cat my-cves.txt | bin/cli list -
is equivalent to bin/cli list CWE-88 CWE-89 CVE-123-456
.
Note that CVEs that are considered to be "incomplete" will not be selected by default. For more information, see Benchmark CVEs.
Many commands accept options specified in a configuration file, config.json
.
By default, bin/cli
looks in the current working directory for
the configuration file. If you want to save your configuration file in an
alternative location, you must specify the path to config.json
using the --config
option on the command line.
Most options that are set in config.json
may be overridden
individually on a per-command basis with identically named
command-line arguments. For example, you can use sources
to provide a
location to download source code for CVEs in config.json
. To override the sources
value in config.json
, you can specify --sources
on the command line.
Configuration files are also used to configure drivers for analysis
tools. For an example of configuring a driver for an analysis tool,
see Configuring the driver for
eslint
, or
browse the README.md files of each driver in
contrib/tools.
This section contains various config.json
examples.
For more information about the allowed contents of a configuration file, see the JSON schema for the Config type.
The configuration file does not need to specify anything, but if
config.json
is present, it should at least be a valid JSON object.
{}
As mentioned above, the .tools
property of the configuration file is
used to configure drivers for analysis tools. The example below configures
a driver for ESLint. You can then benchmark ESLint by running
bin/cli run --tool eslint-default
.
{
"tools": {
"eslint-default": {
"bin": "node",
"args": [
"/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
],
"options": {
"eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
}
}
}
}
It is rarely useful to only have one configured driver for
benchmarking. The sample below configures drivers for two different
versions of ESLint. You can benchmark both versions to compare their performance
by running bin/cli run --tool eslint-default --tool eslint-2019
.
{
"tools": {
"eslint-default": {
"bin": "node",
"args": [
"/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
],
"options": {
"eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
}
},
"eslint-2019": {
"bin": "node",
"args": [
"/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
],
"options": {
"eslintDir": "/home/user-name/analysis-tools/eslint-2019-12-08"
}
}
}
}
The .sources
property of the configuration file controls the
location that the relevant commits of CVEs are downloaded to.
{
"sources": "/home/user-name/ossf-sources"
}