Welcome to the OpenSSF CVE Benchmark project!

The OpenSSF CVE Benchmark consists of code and metadata for over 200 real life CVEs, as well as tooling to analyze the vulnerable codebases using a variety of static analysis security testing (SAST) tools and generate reports to evaluate those tools.

The benchmark addresses two problems that security teams face today when assessing security tools. First, rather than using synthetic test code, the OpenSSF CVE Benchmark uses real life CVEs that have been validated and fixed in open source projects. Using this approach, security tools are tested on real codebases that contain validated real vulnerabilities. Second, by also analyzing the patched versions of the same codebases, false positive rates of these tools can be measured more accurately and based on real validated fixes.

For each CVE in the in the dataset (200+ CVEs so far), the CVE Benchmark determines:

1. Is the security tool able to detect the vulnerability, or does it produce a false negative?
2. When run against the patched codebase, does it recognize the validated patch, or does it produce a false positive?

The CVE Benchmark was developed by the OpenSSF Security Tooling working group and announced at Black Hat Europe 2020. At this time, it covers over 200 JavaScript and TypeScript CVEs. The tooling has integration for benchmarking ESLint, NodeJSScan, and CodeQL. We'd love your help with expanding the dataset, and building integration for more security tools!

Using the CVE Benchmark

Step 1: install the tooling and metadata

Simply clone this repo and use npm to set up the tooling (requires npm ≥ 7 and Node.js ≥ 12):

$ git clone git@github.com:ossf-cve-benchmark/ossf-cve-benchmark.git
$ cd ossf-cve-benchmark
$ npm i
$ npm run-script build

(a warning about "DoubleScrollbar.js" can be ignored, if it appears)

On Linux and macOS, you can now run bin/cli with the appropriate command-line options. On Windows, you need to run bin/cli.cmd.

To confirm that everything is working as it should, run bin/cli -h to display usage and basic help information:

$ bin/cli -h
usage: bin/cli [-h] { ... }

CLI for ossf-cve-benchmark

...

Step 2: run the security tools against the benchmark

The tooling included in the OpenSSF CVE Benchmark will help you evaluate the ability of security tools to generate alerts for the vulnerabilities (CVEs) in the dataset. The benchmark tooling will run the security tools on the source code of the commits associated with CVEs. Using the tools' output, you can then generate benchmark reports that show how the security tools performed. For more information about generating benchmark reports, see Step 3 below.

Before running an analysis, you must install the security tools you wish to benchmark and configure the tool drivers accordingly. A tool driver is a script that acts as the interface between the analysis tools and the CVE Benchmark. For reference information about analysis tool drivers, including information about adding drivers for new tools, see Analysis tools. To see how to set up ESLint to run an analysis, see Generating a benchmark report for eslint

To run an analysis, use the bin/cli run command specifying:

• the identifier of the driver for your analysis tool, as given in config.json, using the --tool option.
• one or more CVEs to run the analysis on. For more information, see Selecting CVEs of interest.

For example, to run an analysis using ESLint over CVE-2018-16492 and CVE-2020-4066 you would use:

$ bin/cli run --tool eslint-default CVE-2018-16492 CVE-2020-4066

where eslint-default is the name of the driver configured for ESLint analysis in the config.json file:

{
  "tools": {
    "eslint-default": {
      "bin": "node",
      "args": [
        "/home/user-name/ossf-cve-benchmark/build/ts/contrib/tools/eslint/src/eslint.js"
      ],
      "options": {
        "eslintDir": "/home/user-name/analysis-tools/eslint-2020-12-08"
      }
    }
  }
}

To run ESLint against all CVEs in the dataset, you can run:

$ bin/cli run --tool eslint-default '*'

Step 3: generate benchmark reports

After running one or more security tools against the dataset (using bin/cli run), you can generate benchmark reports to see how well different security tools performed at detecting vulnerabilities. To do this, you can run bin/cli with the report command, which can generate either an interactive browser-based report, or a text-based report.

Interactive report in your web browser

To view an interactive benchmark report in your browser, you can use the report command with the --kind server option. Just like the run command, it expects you to specify one or more CVEs. You can also pass it '*' to generate a report for all CVEs in the dataset.

For example, to browse a report that compares ESLint to NodeJSScan on all CVEs in the dataset, you would use:

$ bin/cli report --kind server --tool eslint-default --tool nodejsscan '*'

After running this command, you can browse the report at http://localhost:8080.

There are some special ways to select groups of CVEs. For example, if you're interested in a report for just MITRE's Top-25 CWEs, you could run:

$ bin/cli report --kind server --tool eslint-default --tool nodejsscan mitre-cwe-top:25:2020

Text-based reporting

Use --kind txt to create a static text-based report. For example:

$ bin/cli report --kind txt --tool eslint-default CVE-2018-16492 CVE-2020-4066

Status and roadmap

This project has only just started and we'd love your help! The CVE Benchmark currently supports benchmarking three JavaScript security analysis tools on just over 200 CVEs.

In time, we'll work on the following high-level improvements:

• additional CVE data, for both JavaScript and other languages
• support for additional security analysis tools, for both JavaScript and other languages
• more advanced reports

At the meta-level, we expect to soon formalize versioning and release strategies, and to make the workflows for external contributions as seamless as possible.

We don't aim to be ...

• A way of organising and presenting data on all CVEs in existence (such as https://cve.mitre.org)
  • rather, this project hopes that the information in this project makes its way to such views
• A code viewer for browsing alerts (such as a SARIF viewer)
  • rather, this project aims to have a simple analysis result format in order to have a low bar of entry for new analysis tools
• A platform for disclosing new security vulnerabilities (such as https://www.hackerone.com)
  • rather, this project contains simple descriptions of the weaknesses of the above vulnerabilities

We do ❤️ all of the initiatives mentioned above!

Contributing to the project

We welcome contributions to the OpenSSF CVE Benchmark project. If you want to contribute, then please go ahead and open a pull request! See additional details in our contributing guidelines and our FAQ.

About us: the OpenSSF Security Tooling working group

The OpenSSF CVE Benchmark project was initiated by the OpenSSF's Security Tooling working group. The working group contains a wide variety of members; some contribute independently, others join through their roles at organizations like Google, GitHub, Mozilla, Tencent, and OWASP. The working group typically meets (virtually) once every two week; all are welcome.