Cataloging large images is taking too long

[mverleun]

What happened:
When cataloging docker images using a script syft got stuck at one image and continued to consume quite a lot of CPU.
The image that caused trouble is gitlab/gitlab-ce:latest

What you expected to happen:
I would expect that syft would catalog this image as good as any other image.

How to reproduce it (as minimally and precisely as possible):
Download the image and catalog it:

docker pull gitlab/gitlab-ce:latest 
syft -vv -o json --file gitlab_gitlab-ce:latest.sbom.json gitlab/gitlab-ce:latest
...
[0000] DEBUG no new syft update available
[0000] DEBUG image: source=DockerDaemon location=gitlab/gitlab-ce:latest from-lib=stereoscope
[0083] DEBUG image metadata: digest=sha256:a2bf5ef04c22b5530d9c57aa5f20b55601c85fa2393d2d81e120d235f2a39ce4 mediaType=application/vnd.docker.distribution.manifest.v2+json tags=[gitlab/gitlab-ce:latest] from-lib=stereoscope
...
[0150] DEBUG cataloging with "apkdb-cataloger"
[0153] DEBUG discovered 0 packages
[0153] DEBUG cataloging with "go-module-binary-cataloger"
[0164] DEBUG discovered 1490 packages

<stuck here>

Anything else we need to know?:
Scanning the same image with grype goes well.

Environment:

• Output of syft version:
  Application: syft
  Version: 0.32.2
  BuildDate: 2021-12-14T17:38:52Z
  GitCommit: 727b84c
  GitTreeState: clean
  Platform: linux/amd64
  GoVersion: go1.16.11
  Compiler: gc

• OS (e.g: cat /etc/os-release or similar):
  NAME="Ubuntu"
  VERSION="20.04.3 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.3 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

[westonsteimel]

This seems to happen while generating output in the json format specifically, which possibly explains why grype works since syft isn't actually persisting the output in that case

[westonsteimel]

@wagoodman, just a random thought here, but is it possible it's getting into some sort of loop with relationships or something?

[luhring]

That's a good find! When I run with the default output format, it takes a bit of time (this is a large image, and Syft is thorough), but it definitely doesn't get stuck. I believe this started in v0.31.0.

[mverleun]

It is indeed related to the JSON output format. Doing the same scan but this time with output in cyclonedx format goes well.

When reverting to version v0.30.1 the JSON output works fine for this image.

[luhring]

Thanks for confirming!

We're looking at this now. A current theory is a performance cost by our new hash-based method of IDing packages... 🤔

[mverleun]

No, thank you for the quick analysis. I’m quite sure you’ll find a solution. Right now JSON is the only supported format for ‘grype’ and working with a SBOM speeds up analysis quite a bit. So I’ll keep my eye on the future versions.

…

On 15 Dec 2021, at 15:56, Dan Luhring ***@***.***> wrote: Thanks for confirming! We're looking at this now. A current theory is a performance cost by our new hash-based method of IDing packages... 🤔 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#696 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVYZA6J3KSR67XYMTJGD23URCUB3ANCNFSM5KDQO7PA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[wagoodman]

...is it possible it's getting into some sort of loop with relationships

@westonsteimel we've ruled out any infinite loops of relationships for this particular case, and I think the way that we use relationship objects internally would prevent an infinite loops here.

This seems to happen while generating output in the json format specifically...

Indeed! Something that the format.Encode() path for the JSON format and your earlier relationship hunch have in common is their use of Package.ID(), which seems to be the culprit here (as @luhring mentioned earlier):

(I included only the section from the pprof graph that "stood out" since it was pretty large)

We recently stabilized the package ID based on the contents of what is in a package (#363), which leverages the hashstructure lib for dynamically computing the hash of the package object (and we use that hash as the ID of the object). I added in more features around relationship support, which utilizes package.ID() a lot more after the initial change (#607 and #634) . I think this is a worth while feature to keep for reasons os reproducing SBOMs more easily.

After chatting it through with the team I think the best path forward is to implement memoization around the package ID after a certain point in processing. We already have the convention that packages after a certain point in processing do not change, so this change would leverage that assumption.

I'll get this change in ASAP --Thanks for reporting @mverleun and for investigating @westonsteimel @luhring @spiffcs .

[JonZeolla]

We also are seeing an issue with this. A commit yesterday hung for over 2 hours before I stopped it. Unfortunately the logs aren't great quality, but that step essentially runs syft docker-archive:file -o spdx-json --file sbom.thing.spdx.json.