Possibility to easily mirror the grype database in Nexus

[olivierboudet]

What would you like to be added:

I would like to use a private Nexus server to host the database, but it is not very easy to set it up today. I would like to just setup a proxy repo in Nexus, but it is not possible as listing.json file contains absolute URL with grype server hostname.

I think having a listing.json file with relative path to database files and a new config parameter to set the server host prefix can offer the possibility to easily having an internal mirror.

Moreover, it may be necessary to authenticate to such a private mirror, so a username and password may need to be set.

Why is this needed:

I would like to not depend on the availability of the grype infrastructure, and not always download database from internet.

PS : I can send a PR for this, if the proposal is accepted

[kzantow]

Hi @olivierboudet -- Grype should support the standard go proxy environment variables, are you able to use these? E.g. HTTPS_PROXY=https://my.internal.server:<port>

[olivierboudet]

mmmh I think this can't work, Nexus is not a HTTP Proxy.
But I tried this, which is not working : HTTPS_PROXY=https://myhost.com/repository/grype/ grype db update -vvv

[kzantow]

Right, HTTPS_PROXY is for an actual proxy (e.g. SOCKS or whatever go supports). There are multiple caching proxies that could help to accomplish having a local copy of the databases, but that is decidedly different than a mirror. Looking at this in a little more detail: today, Grype supports alternate URLs for the DB listing file with the environment variable GRYPE_DB_UPDATE_URL, so there's at least one option to generate an alternate listing.json file that points to your local network, then just configure Grype to use a this alternate URL (does Nexus possibly have some facility to replace URLs in the listing file for you?).

I can see that currently the listing exists at:
https://toolbox-data.anchore.io/grype/databases/listing.json, and references databases which are accessed at the same base directory (e.g.:
https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v1_2024-02-21T01:23:42Z_8dd50fe43e4e833fc731.tar.gz, https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v5_2023-10-29T01:27:02Z_c490df257cdc4aeae56e.tar.gz). If the URL is a relative URL, it would seem like a pretty small lift to just use the listing URL as the basis.

Just so I understand the ask: you're asking to take this further to then alter the published listing file to use the relative paths, correct? Which would allow you to simply mirror both the listing file and the databases without modification, as you indicated. This certainly seems like a reasonable request to me, but I haven't though through any potential security concerns. I'll bring this up with the team to see if this is something we could update. 👍

[olivierboudet]

Hi @kzantow,
You are correctly understood my idea. I imagine the https://toolbox-data.anchore.io/grype/databases/listing.json file as :

{
    "available": {
        "1": [
            {
                "built": "2024-02-22T01:23:26Z",
                "checksum": "sha256:cc2f950b0db422c444a891a702573d785781dc342fa04d8bec785418ea53f795",
                "url": "./vulnerability-db_v1_2024-02-22T01:23:26Z_355bb785b53ddd8b0d91.tar.gz",
                "version": 1
            },
            {
                "built": "2024-02-21T01:23:42Z",
                "checksum": "sha256:bdb136f881fe7f5edc70be0b602005fe0bbf71bf31a9aea12f5972a4be519bf4",
                "url": "./vulnerability-db_v1_2024-02-21T01:23:42Z_8dd50fe43e4e833fc731.tar.gz",
                "version": 1
            }
        ]
    }
}

If db.update-url is configured with default value (ie. https://toolbox-data.anchore.io/grype/databases/listing.json the database must be resolved as https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v1_2024-02-22T01:23:26Z_355bb785b53ddd8b0d91.tar.gz. But if db.update-url is configured with https://mynexusrepository.com/repository/grype/listing.json the database must be resolved as https://mynexusrepository.com/repository/grype/vulnerability-db_v1_2024-02-22T01:23:26Z_355bb785b53ddd8b0d91.tar.gz

Perhaps it is dangerous to change the behavior for older versions of grype, and it must be necessary to have two files :

• listing.json which will be exactly the same as today
• listing-relative.json which will be the new file with relative paths

[kzantow]

@olivierboudet another option might be to introduce some sort of mirrors configuration specifically for this that Grype understands, which would do something along the lines of extracting the file name portion of URL and trying the configured mirror base URLs to resolve the databases and listing file. Something about this option seems like it could be more easily compromised to me, but since you can already specify the listing URL, I suppose it's not really any different than the risks introduced by that option. This will definitely need some thought before making it into Grype, but I'm pretty sure we can figure out a workable solution here.

[rchaudha]

This feature would make it easier to setup grype repo as a remote repository in Artifactory as well. Rather than us downloading the listing.json and modifying the url to point to our internal Artifactory mirror of the grype repo.