Discussion: Handling differences between NIST "Severity" and Ubuntu "Priority"

[fryz]

Per comment (#3381 (comment)), I'm opening an issue to start a discussion regarding handling differences between NIST Severity and a Data Sources [0] Priorities. CC @knqyf263

Let me know if there's a better forum to have this discussion (mailing list, etc.)

From #3329 (comment):

The priorities assigned to vulnerabilities in Ubuntu are for prioritizing the work of when CVEs will be fixed as opposed to just an assessment of severity, importance or risk. The priority is based on many factors including severity, importance, risk, install base, software configuration, active exploitation and other factors which may adjust the impact of certain vulnerabilities such as Ubuntu's proactive security features. Importantly, these priority levels are distinct from other published severity levels such as CVSS as used in the National Vulnerability Database).

https://people.canonical.com/~ubuntu-security/priority.html#:~:text=CVE%20Priority&text=The%20priority%20is%20based%20on,as%20Ubuntu's%20proactive%20security%20features.

It is different from severity, but this is important information users should be aware of. Technically, it should be shown separately from severity, but here it is to keep things simple. We'd welcome any suggestion to improve.

It seems Trivy is using a leaky abstraction - specifically, it seems to be using the Data Sources "prioritization" mechanism as the Trivy "severity" [1], rather than the NISTs "severity".

I would argue that most users care about the NIST severity and not the data source's priority. Reasons being:

1. Scans are meant to identify images that are vulnerable, and ultimately drive operational workflows to remediate vulnerable images
2. NIST Severity is intended to help prioritize remediation - eg: how quickly should the vulnerable image be upgraded
   • From https://nvd.nist.gov/vuln-metrics/cvss

   • Two common uses of CVSS are calculating the severity of vulnerabilities discovered on one's systems and as a factor in prioritization of vulnerability remediation activities.

3. Ubuntu's Priority is intended to help prioritize fixing the bug - eg: how/when should maintainers schedule the bug to be fixed

Anecdotally, I interact with a handful of enterprise vulnerability SOPs. These SOPs use Trivy (and other scanning tools) to identify vulnerabilities for the purpose of remediating them for compliance/regulatory reasons. The standard for determining the remediation priority is the NIST severity level - not the upstream data sources prioritization (otherwise these policies would need to account for the differences in standards across data sources).

Curious to hear folks thoughts on this. There must be a reason why Trivy uses the data source for scoring, and not the NIST standards, so I'm interested in understanding what that reason is.

Thanks!

[0] in this case, Ubuntu - not sure if other data sources have similar mechanisms that are fed into trivy in a similar fashion

[1] Note the further discussion on #3329:

In the output message, you can write "Priority" instead of "Severity". In this way the user will know it isn't the severity score
But it doesn't look like it can be filtered by --severity. We don't want to add --priority.

[fryz]

Hey @knqyf263,

Bumping this to see if we can discuss changing Trivy's "Severity" to match the NIST "Severity" rather than Ubuntu's "Priority" (or more generally, an upstream datasources definition).

Let me know if there is a better place and/or specific audience that would be better suited for this conversation.

Thanks!

[knqyf263]

Anecdotally, I interact with a handful of enterprise vulnerability SOPs. These SOPs use Trivy (and other scanning tools) to identify vulnerabilities for the purpose of remediating them for compliance/regulatory reasons. The standard for determining the remediation priority is the NIST severity level - not the upstream data sources prioritization (otherwise these policies would need to account for the differences in standards across data sources).

First of all, we're sure the severities from vendors are more accurate. For example, Red Hat scores CVE-2022-3715 as 6.6, even though NVD scores 7.8. It depends on the compile option, the default configuration, etc. NVD doesn't know how the vendor distributes the software. Red Hat evaluates the severity more accurately. So, Trivy definitely should prefer Red Hat scores over NVD. The same can be said for Ubuntu.

The current discussion is "Priority" can be used for "Severity" or not.

Ubuntu's Priority is intended to help prioritize fixing the bug - eg: how/when should maintainers schedule the bug to be fixed

It is different from NVD's severity, but as the document describes, the priority is based on many factors including severity, importance, risk, install base, software configuration, active exploitation and other factors. It is good to know how Ubuntu maintainers are thinking.
As we have said before, while this information is valuable, it could be helpful to treat it as a separate value from severity as it causes confusion.

[huornlmj]

+1, +100 if I could. I am amazed to learn this after using Trivy for more than a year. It's not right to use "Priority" interchangeably with "Severity" - they are two completely different things.

Case in point - Trivy is reporting a new 2023 vulnerability as LOW when it's really HIGH, because Trivy is using "Priority" instead of "Severity"

Ubuntu:22.04 image (hash 5a81c4b8502e) has CVE-2022-3715 (CVSS 7.8 - HIGH). Look at how Trivy reports it:

$ trivy image ubuntu:22.04
ubuntu:22.04 (ubuntu 22.04)

Total: 16 (UNKNOWN: 0, LOW: 10, MEDIUM: 6, HIGH: 0, CRITICAL: 0)

┌─────────────┬────────────────┬──────────┬──────────────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│   Library   │ Vulnerability  │ Severity │    Installed Version     │ Fixed Version │                            Title                            │
├─────────────┼────────────────┼──────────┼──────────────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ bash        │ CVE-2022-3715  │ LOW      │ 5.1-6ubuntu1             │               │ a heap-buffer-overflow in valid_parameter_transform         │
│             │                │          │                          │               │ https://avd.aquasec.com/nvd/cve-2022-3715                   │
├─────────────┼────────────────┤          ├──────────────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤

LOW.

This is because it's Ubuntu's "Priority" field, but Trivy is rewriting that field and calling it the "Severity" field. What's even worse here is that Trivy actually does know that the CVSS is 7.8 but you can only get to this if you use the json format (check the end of the snippet to see the CVSS is 7.8):

"VulnerabilityID": "CVE-2022-3715",
          "PkgID": "bash@5.1-6ubuntu1",
          "PkgName": "bash",
          "InstalledVersion": "5.1-6ubuntu1",
          "Layer": {
            "DiffID": "sha256:59c56aee1fb4dbaeb334aef06088b49902105d1ea0c15a9e5a2a9ce560fa4c5d"
          },
          "SeveritySource": "ubuntu",
          "PrimaryURL": "https://avd.aquasec.com/nvd/cve-2022-3715",
          "DataSource": {
            "ID": "ubuntu",
            "Name": "Ubuntu CVE Tracker",
            "URL": "https://git.launchpad.net/ubuntu-cve-tracker"
          },
          "Title": "a heap-buffer-overflow in valid_parameter_transform",
          "Description": "A flaw was found in the bash package, where a heap-buffer overflow can occur in valid parameter_transform. This issue may lead to memory problems.",
          "Severity": "LOW",
          "CweIDs": [
            "CWE-787"
          ],
          "CVSS": {
            "nvd": {
              "V3Vector": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
              "V3Score": 7.8
            },
            "redhat": {
              "V3Vector": "CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:U/C:L/I:L/A:H",
              "V3Score": 6.6
            }
          },

This was an accidental finding, albeit one from a colleague who had other doubts as to Trivy's accuracy. I believe this is the systemic problem that is causing such doubt amongst Trivy users.

Please fix this flaw if you want people to use what is in my mind a fabulous tool, in confidence. In the interim please provide a switch so that people using Trivy can opt to have the CVSS.V3Score field depict the actual severity when using table view.

Issue #4835

[stian-svedenborg-gul-katt]

Indeed letting the user decide which rating to use, (or show both) would make trivvy much easier to use.

I agree with the argument that distribution maintainers will likely understand better what real angles of attack are available for a given vulnerability. However, they do not know whether a particular issue is more exposed on any given system instance because of local concerns.

This discussion has been going on for about a year now, are you ready to conclude on some action?

[dhensby]

There seems to be a consensus that the "Priority" field (for Ubuntu, at least) is not interchangeable with "Severity", and that using it as such is misleading.

I also understand the point that vendors may re-score a vulnerability (more accurately) and that this should be taken into account. However, if the APIs vendors provide for these vulnerabilities does not have a consistent way to declare a Severity, then perhaps it is better if the reported CVSS score (even if it's adjusted by the vendor), should be used by trivy to determine the Severity, rather than using a potentially mis-attributed field.