Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add distro information to package URLs for OS packages #754

Merged
merged 4 commits into from Jan 19, 2022
Merged

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Jan 18, 2022

This PR does primarily enhances up support for OS-related packages to include distro version information in any pURL generated:

- pkg:alpine/name@1.0.1?arch=amd64
+ pkg:alpine/name@1.0.1?arch=amd64&distro=alpine-3.4.6

The ID and VersionID are encoded into the pURL as a distro get-param; see the pURL spec for more info.

Additionally this PR:

  • migrates the existing package URL functionality to the pkg package (since this semantically represents packages)
  • beefs up testing around pURL generation to ensure that we are exercising all package types
  • during this work it was found that the NPM metadata filename within pkg was inconsistent --the filename was updated

Related to anchore/grype#395

Note: this PR cannot be merged until #753 is merged and this PR is rebased.

@wagoodman wagoodman added the enhancement New feature or request label Jan 18, 2022
@wagoodman wagoodman requested a review from a team January 18, 2022 22:11
@wagoodman wagoodman self-assigned this Jan 18, 2022
@github-actions
Copy link

github-actions bot commented Jan 18, 2022

Benchmark Test Results

Benchmark results from the latest changes vs base branch
name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.26ms ± 1%    1.70ms ± 1%  +35.00%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            3.20ms ± 7%    3.99ms ± 1%  +24.71%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.11ms ± 2%    1.37ms ± 4%  +22.97%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         864µs ± 4%    1061µs ± 1%  +22.87%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     986µs ± 3%    1240µs ± 2%  +25.83%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      907µs ± 3%    1122µs ± 1%  +23.66%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      14.8ms ± 1%    18.0ms ± 1%  +21.63%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.36ms ± 2%    1.72ms ± 1%  +26.66%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          2.14µs ± 1%    2.42µs ± 1%  +12.73%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               253kB ± 0%     253kB ± 0%   +0.11%  (p=0.032 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            1.06MB ± 0%    1.06MB ± 0%     ~     (p=0.841 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     253kB ± 0%     253kB ± 0%     ~     (p=0.222 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         207kB ± 0%     208kB ± 0%   +0.14%  (p=0.016 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     254kB ± 0%     254kB ± 0%   +0.24%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      235kB ± 0%     236kB ± 0%   +0.14%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.78MB ± 0%    3.78MB ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.30MB ± 0%    1.30MB ± 0%     ~     (p=0.841 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            560B ± 0%      560B ± 0%     ~     (all equal)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2               6.33k ± 0%     6.33k ± 0%     ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2             21.4k ± 0%     21.4k ± 0%     ~     (p=0.524 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     7.26k ± 0%     7.26k ± 0%     ~     (p=0.643 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         5.34k ± 0%     5.34k ± 0%     ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                     7.07k ± 0%     7.10k ± 0%   +0.51%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                      6.80k ± 0%     6.82k ± 0%   +0.22%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                       74.7k ± 0%     74.7k ± 0%     ~     (p=0.167 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      7.36k ± 0%     7.36k ± 0%   +0.07%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            13.0 ± 0%      13.0 ± 0%     ~     (all equal)

Copy link
Contributor

@kzantow kzantow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you think this is ready, 👍

},
metadata: DpkgMetadata{
Package: "p",
Source: "s",
Version: "v",
Architecture: "a",
},
expected: "pkg:deb/ubuntu/p@v?arch=a",
expected: "pkg:deb/ubuntu/p@v?arch=a&distro=ubuntu-16.04",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question, though: should these be more like pkg:deb/ubuntu/p@v?arch=a&distro=16.04 without the ubuntu-? (After looking again at: https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#deb )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I'm not certain. The spec only has one "good" example with this here: https://github.com/package-url/purl-spec/blob/1b1e9b2afa6de1c855225fb08cb46878ad653925/PURL-SPECIFICATION.rst#some-purl-examples

pkg:deb/debian/curl@7.50.3-1?arch=i386&distro=jessie
pkg:rpm/fedora/curl@7.50.3-1.fc25?arch=i386&distro=fedora-25

The debian distro code-name is a bad idea for us, so I didn't approach it that way... so the fedora example was used as a basis. That being said, there is essentially 0 verbiage on correct usage of distro even though it is a suggested qualifier (that we need).

References:

Since there is little direction from the spec I'm open to other options here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From an offline conversation: we'll leave this as is for now since it seems more semantically correct to the term "distro" but are open to changing in the future.

Base automatically changed from add-php-metadata to main January 19, 2022 16:42
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
Copy link
Contributor Author

(force pushed to rebase from main now that #753 is merged)

@wagoodman wagoodman marked this pull request as draft January 19, 2022 22:11
@wagoodman wagoodman marked this pull request as ready for review January 19, 2022 22:11
@wagoodman
Copy link
Contributor Author

wagoodman commented Jan 19, 2022

looks like the codeql workflow hasn't kicked in after a long while (hours)

Screen Shot 2022-01-19 at 5 11 14 PM

This could be due to the fact that the base branch was originally not main but was automatically switched when the reference PR merged.

Changing the PR from draft state and back didn't do the trick. Neither did switching the base branch to something else and back to main.

Also it seems like all pull request events are triggering for this PR at all... maybe a github actions bug?

@wagoodman wagoodman changed the base branch from main to add-fingerprint January 19, 2022 22:14
@wagoodman wagoodman changed the base branch from add-fingerprint to main January 19, 2022 22:14
@wagoodman wagoodman enabled auto-merge (squash) January 19, 2022 22:17
@wagoodman
Copy link
Contributor Author

I'm going to close and reopen this PR to hopefully trigger the pull_request events for this PR.

@wagoodman wagoodman closed this Jan 19, 2022
auto-merge was automatically disabled January 19, 2022 22:24

Pull request was closed

@wagoodman wagoodman reopened this Jan 19, 2022
@wagoodman
Copy link
Contributor Author

That's more like it 💯

Screen Shot 2022-01-19 at 5 24 32 PM

@wagoodman wagoodman enabled auto-merge (squash) January 19, 2022 22:25
@wagoodman wagoodman merged commit 1350d6c into main Jan 19, 2022
@wagoodman wagoodman deleted the improve-purls branch January 19, 2022 22:30
fengshunli pushed a commit to fengshunli/syft that referenced this pull request Jan 24, 2022
* rename npm metadata struct

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve os package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve language package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* wire up composer pURL method

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: fsl <1171313930@qq.com>
spiffcs pushed a commit that referenced this pull request Jan 25, 2022
* rename npm metadata struct

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve os package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve language package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* wire up composer pURL method

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman changed the title Improve package URL support Add distro information to package URLs for OS packages Jan 28, 2022
jonasagx pushed a commit to jonasagx/syft that referenced this pull request Jan 28, 2022
* rename npm metadata struct

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve os package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve language package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* wire up composer pURL method

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
* rename npm metadata struct

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve os package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* improve language package URLs

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* wire up composer pURL method

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants