Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve SPDX decoding functionality #738

Merged
merged 48 commits into from Feb 9, 2022
Merged

Improve SPDX decoding functionality #738

merged 48 commits into from Feb 9, 2022

Conversation

kzantow
Copy link
Contributor

@kzantow kzantow commented Jan 7, 2022

Adds the ability to ingest SPDX JSON and Tag-Value formats and convert them to internal Syft data model.

This will be the basis for: anchore/grype#395

Note: there is a branch with a quick hack of SBOM input for Syft

@github-actions
Copy link

github-actions bot commented Jan 7, 2022

Benchmark Test Results

Benchmark results from the latest changes vs base branch
name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.55ms ± 5%    1.56ms ± 1%    ~     (p=0.310 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            3.57ms ± 5%    3.63ms ± 8%    ~     (p=0.841 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.24ms ± 1%    1.22ms ± 1%  -1.30%  (p=0.032 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2        1.00ms ± 8%    0.95ms ± 2%  -5.17%  (p=0.032 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                    1.18ms ± 4%    1.11ms ± 2%  -5.79%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                     1.00ms ± 2%    1.00ms ± 3%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      16.3ms ± 2%    15.7ms ± 1%  -3.18%  (p=0.016 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.52ms ± 3%    1.48ms ± 1%  -2.83%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          2.14µs ± 1%    2.17µs ± 3%    ~     (p=0.452 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               252kB ± 0%     253kB ± 0%    ~     (p=0.056 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            1.06MB ± 0%    1.06MB ± 0%    ~     (p=0.421 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     253kB ± 0%     253kB ± 0%    ~     (p=0.841 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         208kB ± 0%     208kB ± 0%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     254kB ± 0%     254kB ± 0%    ~     (p=0.421 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      236kB ± 0%     236kB ± 0%    ~     (p=0.151 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.78MB ± 0%    3.78MB ± 0%    ~     (p=0.151 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.30MB ± 0%    1.30MB ± 0%    ~     (p=0.643 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            560B ± 0%      560B ± 0%    ~     (all equal)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               6.33k ± 0%     6.33k ± 0%    ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             21.4k ± 0%     21.4k ± 0%    ~     (p=0.246 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     7.26k ± 0%     7.26k ± 0%    ~     (p=0.095 n=5+4)
ImagePackageCatalogers/javascript-package-cataloger-2         5.36k ± 0%     5.36k ± 0%    ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     7.11k ± 0%     7.11k ± 0%    ~     (all equal)
ImagePackageCatalogers/rpmdb-cataloger-2                      6.83k ± 0%     6.83k ± 0%    ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       74.7k ± 0%     74.7k ± 0%    ~     (p=0.794 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      7.38k ± 0%     7.38k ± 0%    ~     (p=1.000 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            13.0 ± 0%      13.0 ± 0%    ~     (all equal)

@kzantow kzantow changed the title Add SPDX decoding [Very WiP] Add SPDX decoding Jan 10, 2022
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
@kzantow kzantow changed the title [Very WiP] Add SPDX decoding [WiP] Add SPDX decoding Jan 26, 2022
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
@kzantow kzantow requested a review from a team February 7, 2022 17:54
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Keith Zantow <kzantow@gmail.com>
s := &sbom.SBOM{
Artifacts: sbom.Artifacts{
PackageCatalog: pkg.NewCatalog(),
FileMetadata: map[source.Coordinates]source.FileMetadata{},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: idiomatic go comment

Suggested change
FileMetadata: map[source.Coordinates]source.FileMetadata{},
FileMetadata: make(map[source.Coordinates]source.FileMetadata),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, will remember this and follow on a fix later :)

for _, typ := range f.FileType {
switch FileType(typ) {
case ImageFileType:
meta.MIMEType = "image/"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: is is valid to have partial mimetypes like this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really, but we don't have any additional info, this was the lossiness I was referring to

Copy link
Contributor

@wagoodman wagoodman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're at the finish line 🏁 Awesome work!

It would be a good idea to hold off merging this until we finalize the grype branch that uses this work and verify that everything integrates as expected 👍

@kzantow kzantow merged commit 76f8205 into main Feb 9, 2022
@kzantow kzantow deleted the add-spdx-decoder-take-2 branch February 9, 2022 19:11
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants