eslint-example.md

Example: Generating a benchmark report for eslint

This example shows you how to:

• Install the eslint analysis tool.
• Configure the driver for eslint.
• Run eslint on the source code of selected CVEs using bin/cli run.
• View a benchmark report for the analysis run using bin/cli report.

Prerequisites

Before working through this example, you must install the CVE Benchmark tooling. For more information, see Using the CVE Benchmark.

Installing eslint

You must install each analysis tool that you want to benchmark with bin/cli run. Some drivers are provided with installer scripts that install their backing analysis tool, while other drivers require manual installation steps. When adding support for a new analysis tool, we encourage you to provide an installer.

To see which tools are available, and which have installers, run bin/cli tools:

$ bin/cli tools
Configured tools in /home/user-name/ossf-cve-benchmarking/config.json:
...

Available driver installers:
- ...
- contrib/tools/eslint/installers/install.cmd
- contrib/tools/eslint/installers/install.sh
- ...

Available driver READMEs:
- ...
- contrib/tools/eslint/README.md
- ...

Configured:
...

The output shows that eslint is available with an installer. Run the installer, specifying a directory to install eslint in:

$ contrib/tools/eslint/installers/install.sh /home/user-name/analysis-tools/eslint-2020-12-08
The eslint tool has been installed. Add the fragment below to a config.json file:

{
  ...
  "tools": {
    ...
    "eslint-default": {
      "bin": "node",
      "args": [
        "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
      ],
      "options": {
        "eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
      }
    }
    ...
  }
  ...
}

eslint has now been successfully installed. bin/cli does not interact directly with eslint, but rather through a driver that executes eslint on selected code and converts the results to a form that benchmark reports can be generated from. So, to complete the setup, you must also configure the driver.

Configuring the driver for eslint

To configure the driver for an analysis tool, you need to add a tools entry to your local config.json file.

For eslint, the installer script displays the appropriate snippet at the end of its output:

{
  "tools": {
    "eslint-default": {
        "bin": "node",
        "args": [
            "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js",
        ]
        
    }
    "options": {
      "eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
    }
  }
}

This snippet includes the identifier eslint-default that you need to specify with the --tool option on the command line. After updating your config.json file with this snippet, check that eslint has been successfully configured by running bin/cli tools again:

$ bin/cli tools
Configured:
- eslint-default
...

The identifier eslint-default is now listed as configured for use with bin/cli.

Running eslint

After completing the steps above, you can use bin/cli run to run an analysis with eslint over one or more benchmark CVEs. In this example we'll select two CVEs, CVE-2018-16492 and CVE-2020-4066.

To run the analysis with eslint, you must specify the name of the identifier used in your config.json file, eslint-default, with the --tool option:

$ bin/cli run --tool eslint-default CVE-2018-16492 CVE-2020-4066
...

For more information about specifying data to analyze with your tool, see Selecting CVEs of interest.

After running the command, results files are generated in JSON format. By default, the files are written to disk in work/results/eslint-default_....json.

Creating reports for the runs

Now that you have data for some runs, you can view a benchmark report using the bin/cli report command.

To start a report server that you can view in your browser, run the following:

$ bin/cli report --kind server --tool eslint-default CVE-2018-16492 CVE-2020-4066

Navigate to http://localhost:8080 in your browser to view the report.