Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce a single SBOM document #606

Merged
merged 16 commits into from Nov 5, 2021
Merged

Introduce a single SBOM document #606

merged 16 commits into from Nov 5, 2021

Conversation

wagoodman
Copy link
Contributor

This PR replaces poweruser.JSONDocumentConfig with sbom.SBOM. Specifically this datastructure is being promoted as a first-class object to be used internally for tasking, presentation, and processing concerns. This is proposed in #555 (comment) but closely affects #554 and #556.

The main proposal of the PR is the following datastructure:

syft/syft/sbom/sbom.go

Lines 10 to 23 in a26e9e4

type SBOM struct {
Artifacts Artifacts
Source source.Metadata
}
type Artifacts struct {
PackageCatalog *pkg.Catalog
FileMetadata map[source.Location]source.FileMetadata
FileDigests map[source.Location][]file.Digest
FileClassifications map[source.Location][]file.Classification
FileContents map[source.Location]string
Secrets map[source.Location][]file.SearchResult
Distro *distro.Distro
}

All other changes attempt to promote usage of this data structure over passing parts of an SBOM around, for example... replacing:

func Encode(catalog *pkg.Catalog, metadata *source.Metadata, dist *distro.Distro, scope source.Scope, option format.Option) ([]byte, error) { ... }

with:

func Encode(s sbom.SBOM, option format.Option) ([]byte, error) { ... }

Notes:

@wagoodman wagoodman requested a review from a team October 29, 2021 16:40
@wagoodman wagoodman self-assigned this Oct 29, 2021
@github-actions
Copy link

github-actions bot commented Oct 29, 2021

Benchmark Test Results

Benchmark results from the latest changes vs base branch
name                                                   old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2          1.37ms ± 3%    1.67ms ± 2%  +21.98%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        3.29ms ± 6%    3.88ms ± 6%  +18.15%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     821µs ± 2%     996µs ± 3%  +21.35%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 893µs ± 4%    1077µs ± 2%  +20.60%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  848µs ± 1%    1013µs ± 2%  +19.42%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  11.8ms ± 3%    14.2ms ± 1%  +20.49%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.22ms ± 7%    1.55ms ± 3%  +26.69%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2       654ns ± 4%     783ns ± 3%  +19.65%  (p=0.008 n=5+5)

name                                                   old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           248kB ± 0%     247kB ± 0%   -0.38%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        1.11MB ± 0%    1.11MB ± 0%   -0.33%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     199kB ± 0%     197kB ± 0%   -0.68%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 228kB ± 0%     228kB ± 0%   -0.39%  (p=0.016 n=5+4)
ImagePackageCatalogers/rpmdb-cataloger-2                  222kB ± 0%     221kB ± 0%   -0.48%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  3.24MB ± 0%    3.24MB ± 0%   -0.13%  (p=0.032 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.29MB ± 0%    1.29MB ± 0%   -0.08%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2        336B ± 0%      336B ± 0%     ~     (all equal)

name                                                   old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           6.82k ± 0%     6.79k ± 0%     ~     (p=0.079 n=4+5)
ImagePackageCatalogers/python-package-cataloger-2         26.3k ± 0%     26.3k ± 0%   -0.27%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     5.19k ± 0%     5.17k ± 0%   -0.46%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 6.67k ± 0%     6.65k ± 0%   -0.36%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  6.56k ± 0%     6.53k ± 0%   -0.37%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                   59.0k ± 0%     58.9k ± 0%   -0.20%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                  7.74k ± 0%     7.71k ± 0%   -0.30%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2        9.00 ± 0%      9.00 ± 0%     ~     (all equal)

@wagoodman wagoodman marked this pull request as ready for review November 2, 2021 14:23
Copy link
Contributor

@spiffcs spiffcs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass.

I like the new SBOM struct and how it just encapsulates current information. Is there a future where the underlying Artifacts struct is changed as we build into developing relationships between everything?

Also small comments on data loss across the new outputs of the golden snapshots. I wasn't sure if we wanted to start excluding the information, or if it got lost as a consequence of moving to the new encode architecture.

@wagoodman wagoodman force-pushed the single-sbom-document branch 2 times, most recently from bf6a2cd to 4757c25 Compare November 2, 2021 17:39
@wagoodman
Copy link
Contributor Author

@spiffcs

Is there a future where the underlying Artifacts struct is changed as we build into developing relationships between everything?

Indeed! I have some of that drafted out in #607, specifically:

Relationships []artifact.Relationship

This is where the relationships object gets promoted to in the end (the sbom.SBOM struct).

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@spiffcs spiffcs force-pushed the single-sbom-document branch 5 times, most recently from dabefda to 19b9013 Compare November 3, 2021 18:14
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
@spiffcs spiffcs requested a review from a team November 4, 2021 16:47
Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Copy link
Contributor

@luhring luhring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice work! 👏

I had just one question from curiosity

cmd/packages.go Show resolved Hide resolved
@wagoodman
Copy link
Contributor Author

@spiffcs thanks a ton for solving what ended up being a caching issue I was having --I was starting to go a little crazy there 🙌

@wagoodman wagoodman merged commit bb0f35b into main Nov 5, 2021
@wagoodman wagoodman deleted the single-sbom-document branch November 5, 2021 14:05
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
* [wip] single sbom doc

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix more tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix linting

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update cli tests

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* remove scope in import path

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* swap SPDX tag-value formatter to single sbom document

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* bust CLI cache

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* update fixture to byte diff

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* byte for byte

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* bust the cache

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* who needs cache

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* add jar for testing

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* no more bit flips

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* update apk with the delta for image and directory cases

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

* restore cache workflow

Signed-off-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>

Co-authored-by: Christopher Angelo Phillips <christopher.phillips@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants