Merge v0.9.2-dev to master (#946)

* Improve the message and description for vulnerability_data_unavailable and stale_feed_data triggers in the vulnerabilities gate. Fixes #879 Signed-off-by: Zach Hill <zach@anchore.com> * Bump version numbers for 0.9.1 Signed-off-by: Robert Prince <robert.prince@anchore.com> * Multiple policy bundle dirs (#862) * Allow for localconfig to read policy bundles from multiple dirs. Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Expect fully-qualifed policy bundle dirs. Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Reload policy bundle from file whenever a new bundle dir is added. Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Linting Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Linting, again. Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Linting commas Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Fix test. Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Code review comments, add some extra logging and another test. Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Linting Signed-off-by: Daniel Palmer <dan.palmer@anchore.com> * Fix method name to match parent class Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * Removed ui from swagger url Signed-off-by: Zane Burstein <zane.burstein@anchore.com> * Add ability to support multiple grant types for the oauth client Signed-off-by: Zach Hill <zach@anchore.com> * Update Dockerfile to use UBI 8.3. Fixes #888 Signed-off-by: Zach Hill <zach@anchore.com> * Update CHANGELOG.md for 0.9.1 Signed-off-by: Zach Hill <zach@anchore.com> * Fix confusing typo in changelog Signed-off-by: Zach Hill <zach@anchore.com> * Update syft to v0.12.5 Signed-off-by: Dan Luhring <dan.luhring@anchore.com> * add bundles/ dir to anchore_service_dir Signed-off-by: Brady Todhunter <bradyt@anchore.com> * Updates to vulnerability listing dedup logic - prioritize vulnerabilities from other namespaces over nvd out vulnerabilities - filter duplicates Fixes #893 Signed-off-by: Swathi Gangisetty <swathi@anchore.com> * Set the python package location according to the package key, which is the absolute path (#895) Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * Update the scanner config method in policy engine for providing overr… (#896) * Update the scanner config method in policy engine for providing overridable functions for vuln and cpe results. Adds use of that in the vulnerability policy gate Signed-off-by: Zach Hill <zach@anchore.com> * first draft at a dedup pass Signed-off-by: Swathi Gangisetty <swathi@anchore.com> * Try to load Policy Engine ImageCpes from syft generated cpes, with fallback to fuzzy matching Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * Add unit test for loader paths Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * include update and meta into cpe comparison Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * fix return type Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * Unit tests for cpe comparisons used for vulnerability dedup Signed-off-by: Swathi Gangisetty <swathi@anchore.com> * Downgrade empty content log message to debug level Signed-off-by: Swathi Gangisetty <swathi@anchore.com> * tests: change previous invalid schema for integers/floats According to draft-6 which the new jsonschema supports 1.0 is considered an integer. Relevant doc from the draft: In draft-04, "integer" is listed as a primitive type and defined as “a JSON number without a fraction or exponent part”; in draft-06, "integer" is not considered a primitive type and is only defined in the section for keyword "type" as “any number with a zero fractional part”; 1.0 is thus not a valid "integer" type in draft-04 and earlier, but is a valid "integer" type in draft-06 and later; note that both drafts say that integers SHOULD be encoded in JSON without fractional parts Link https://json-schema.org/draft-06/json-schema-release-notes.html Signed-off-by: Alfredo Deza <adeza@anchore.com> (cherry picked from commit 2f859a1) * requirements: bump jsonschema to avoid legacy validator import issues Signed-off-by: Alfredo Deza <adeza@anchore.com> (cherry picked from commit 99dcb10) * Update syft to 0.12.7 to fix analysis failure due to syft parsing issue. Fixes #910 Signed-off-by: Zach Hill <zach@anchore.com> * Update cryptography lib to 3.3.2 from 3.3.1. Fixes #909 Signed-off-by: Zach Hill <zach@anchore.com> * add package filtering by relationships Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * Fix the client metadata merge process during oauth init. Fixes #931 Signed-off-by: Zach Hill <zach@anchore.com> * Bump version Signed-off-by: Robert Prince <robert.prince@anchore.com> * Add default admin pw to e2e test values file Signed-off-by: Robert Prince <robert.prince@anchore.com> * Make sure to return content correctly for manifest and dockerfile content types Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * [docs] 0.9.2 release notes and changelog updates. includes missing release notes for 0.9.1 (#939) Updates CHANGELOG for 0.9.2 and adds 0.9.1 and 0.9.2 release notes Also fixes ordering problem in release notes page Signed-off-by: Zach Hill <zach@anchore.com> * Update Quickstart Docker-Compose image tag to v0.9.2 Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * Iterate API patch version 0.1.16->0.1.17 Signed-off-by: Samuel Dacanay <sam.dacanay@anchore.com> * Add distro mapping from "redhat" to "rhel" for vuln matching Signed-off-by: Zach Hill <zach@anchore.com> * Adds distro mapper in import path to ensure rhel instead of redhat distro name Signed-off-by: Zach Hill <zach@anchore.com> * Fix integration tests that used redhat as a negative test example Signed-off-by: Zach Hill <zach@anchore.com> Co-authored-by: Zach Hill <zach@anchore.com> Co-authored-by: Dan Palmer <dan.palmer@anchore.com> Co-authored-by: Samuel Dacanay <sam.dacanay@anchore.com> Co-authored-by: Zane Burstein <zane.burstein@anchore.com> Co-authored-by: Dan Luhring <dan.luhring@anchore.com> Co-authored-by: Brady Todhunter <bradyt@anchore.com> Co-authored-by: Swathi Gangisetty <swathi@anchore.com> Co-authored-by: Alfredo Deza <adeza@anchore.com> Co-authored-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: Alex Goodman <wagoodman@users.noreply.github.com>
anchore · Mar 11, 2021 · 0bb613b · 0bb613b
1 parent e1e290e
commit 0bb613b
Show file tree

Hide file tree

Showing 41 changed files with 528 additions and 342 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,15 @@
 # Changelog
 
+## 0.9.2
+
++ Fixed - Fixes empty string value for "metadata" field which should be empty array in response for GET /images/{digest}/metadata/dockerfile when no actual dockerfile is presented. Fixes #937
++ Fixed - Fixes oauth2_clients table upgrade to include all needed keys in client_metadata field. Fixes #931
++ Fixed - Updates syft to 0.13.1 and adds filtering of packages by new 'relationship' field to remove duplicate packages that are application packages provided by distro packages managers (e.g. RPMs that install python eggs, will only use the RPM version). Fixes #460
++ Fixed - Updates syft to 0.12.7 to fix analysis failure due to malformed python egg files. Fixes #910
++ Fixed - Updates cryptography version from 3.3.1 to 3.3.2. Fixes #909
++ Fixed - Updates jsonschema version to avoid legacy validator import issues.
+
+
 ## 0.9.1
 
 NOTE: To ensure that Anchore Engine cannot be accidentally deployed with a weak default password for the admin user, this release includes

diff --git a/Dockerfile b/Dockerfile
@@ -46,7 +46,7 @@ RUN set -ex && \
 
 RUN set -ex && \
     echo "installing Syft" && \
-    curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /build_output/deps v0.12.5
+    curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /build_output/deps v0.13.1
 
 # stage RPM dependency binaries
 RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm && \
@@ -62,7 +62,7 @@ FROM registry.access.redhat.com/ubi8/ubi:8.3 as anchore-engine-final
 
 ARG CLI_COMMIT
 ARG ANCHORE_COMMIT
-ARG ANCHORE_ENGINE_VERSION="0.9.1"
+ARG ANCHORE_ENGINE_VERSION="0.9.2"
 ARG ANCHORE_ENGINE_RELEASE="r0"
 
 # Copy skopeo artifacts from build step

diff --git a/anchore_engine/analyzers/syft/__init__.py b/anchore_engine/analyzers/syft/__init__.py
@@ -1,12 +1,37 @@
 import collections
 
-from anchore_engine.analyzers.utils import defaultdict_to_dict, content_hints
+from anchore_engine.analyzers.utils import defaultdict_to_dict, content_hints, dig
 from anchore_engine.clients.syft_wrapper import run_syft
 from .handlers import modules_by_artifact_type, modules_by_engine_type
 
 
-def filter_artifacts(artifact):
-    return artifact["type"] in modules_by_artifact_type
+def filter_relationships(relationships, **kwargs):
+    def filter_fn(relationship):
+        for key, expected in kwargs.items():
+            if relationship[key] != expected:
+                return False
+        return True
+
+    return [r for r in relationships if filter_fn(r)]
+
+
+def filter_artifacts(artifacts, relationships):
+    def filter_fn(artifact):
+        # syft may do more work than what is supported in engine, ensure we only include artifacts
+        # of select package types.
+        if artifact["type"] not in modules_by_artifact_type:
+            return False
+
+        # some packages are owned by other packages (e.g. a python package that was installed
+        # from an RPM instead of with pip), filter out any packages that are not "root" packages.
+        if filter_relationships(
+            relationships, child=dig(artifact, "id"), type="ownership-by-file-overlap"
+        ):
+            return False
+
+        return True
+
+    return [a for a in artifacts if filter_fn(a)]
 
 
 def catalog_image(imagedir):
@@ -42,7 +67,10 @@ def convert_syft_to_engine(all_results):
     # take a sub-set of the syft findings and invoke the handler function to
     # craft the artifact document and inject into the "raw" analyzer json
     # document
-    for artifact in filter(filter_artifacts, all_results["artifacts"]):
+    for artifact in filter_artifacts(
+        all_results["artifacts"],
+        dig(all_results, "artifactRelationships", force_default=[]),
+    ):
         handler = modules_by_artifact_type[artifact["type"]]
         handler.translate_and_save_entry(findings, artifact)
 

diff --git a/anchore_engine/apis/oauth.py b/anchore_engine/apis/oauth.py
@@ -180,18 +180,27 @@ def merge_client_metadata(found_meta: dict, expected_metadata: dict) -> dict:
     """
     Merge the client metadata from what is found and what is needed to create a single metadata record.
 
-    Typically this is just a merging of the grant_types lists.
+    This includes a merge of the grant_types via a union operation, and replacement of any conflicting keys in the found_meta with values from expected_meta.
 
-    :param found_client:
-    :param client_metadata:
+    :param found_meta: The metadata dict from the existing record
+    :param expected_metadata: The metadata dict to merge in
     :return: dict of merged information
     """
-    merged = copy.copy(found_meta) if found_meta else {}
 
     # Merge the new grant types in, defaulting to empty grant lists if not found
-    grants = merged.get(CLIENT_GRANT_KEY, [])
-    grants.extend(expected_metadata.get(CLIENT_GRANT_KEY, []))
-    merged[CLIENT_GRANT_KEY] = list(set(grants))
+    if found_meta is None:
+        found_meta = {}
+
+    found_grants = set(found_meta.get(CLIENT_GRANT_KEY, []))
+    new_grants = set(expected_metadata.get(CLIENT_GRANT_KEY, []))
+    new_grants = new_grants.union(found_grants)
+
+    # Create a copy to ensure we don't modify the state of anything passed in
+    merged = copy.copy(found_meta)
+
+    # Merge in the non-grant keys
+    merged.update(expected_metadata)
+    merged[CLIENT_GRANT_KEY] = list(new_grants)
 
     return merged
 
@@ -210,10 +219,18 @@ def setup_oauth_client(found: OAuth2Client, to_merge: OAuth2Client) -> OAuth2Cli
 
         # Ensure the client record has the right set of grant types, not one grant per client, since we have a single client_id
         found_meta = found.client_metadata
-
         merged = merge_client_metadata(found_meta, to_merge.client_metadata)
+
+        # Try a simple set first, if it doesn't work, the update the dict content directly. This is necessary
+        # due to the implementation of the client_metadata property
         found.set_client_metadata(merged)
 
+        # Have to clear because the "set_client_metadata" doesn't work properly once the data is initialized.
+        # So use an in-place update, and the 'merged' state will replace all the state.
+        if found.client_metadata != merged:
+            found.client_metadata.clear()
+            found.client_metadata.update(merged)
+
         logger.info(
             "Updated %s OAuth client record with grants %s",
             found.client_id,

diff --git a/anchore_engine/services/analyzer/imports.py b/anchore_engine/services/analyzer/imports.py
@@ -132,12 +132,17 @@ def process_import(
 
         timer = time.time()
 
+        distro = syft_packages.get("distro", {}).get("name")
+        # Map 'redhat' distro to 'rhel' distro for consistency between internal metadata fetch from squashtar and the syft implementation used for import
+        if distro == "redhat":
+            distro = "rhel"
+
         # Move data from the syft sbom into the analyzer output
         analyzer_report = {
             "analyzer_meta": {
                 "analyzer_meta": {
                     "base": {
-                        "DISTRO": syft_packages.get("distro", {}).get("name"),
+                        "DISTRO": distro,
                         "DISTROVERS": syft_packages.get("distro", {}).get("version"),
                         "LIKEDISTRO": syft_packages.get("distro", {}).get("idLike"),
                     }

diff --git a/anchore_engine/services/apiext/swagger/swagger.yaml b/anchore_engine/services/apiext/swagger/swagger.yaml
@@ -1,7 +1,7 @@
 swagger: "2.0"
 info:
   description: "This is the Anchore Engine API. Provides the primary external API for users of the service."
-  version: "0.1.16"
+  version: "0.1.17"
   title: "Anchore Engine API Server"
   contact:
     email: "nurmi@anchore.com"
@@ -5011,6 +5011,10 @@ definitions:
         "$ref": "#/definitions/ImportDescriptor"
       schema:
         "$ref": "#/definitions/ImportSchema"
+      artifactRelationships:
+        items:
+          "$ref": "#/definitions/ImportPackageRelationship"
+        type: array
     additionalProperties: true
     type: object
   ImportDescriptor:
@@ -5022,7 +5026,7 @@ definitions:
         type: string
       version:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportDistribution:
     required:
@@ -5036,7 +5040,7 @@ definitions:
         type: string
       idLike:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportPackageLocation:
     required:
@@ -5046,7 +5050,7 @@ definitions:
         type: string
       layerID:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportPackage:
     required:
@@ -5059,6 +5063,8 @@ definitions:
       - cpes
       - metadataType
     properties:
+      id:
+        type: string
       name:
         type: string
       version:
@@ -5088,9 +5094,8 @@ definitions:
         type: string
       metadata:
         type: object
-    additionalProperties: false
+    additionalProperties: true
     type: object
-
   ImportSchema:
     required:
       - version
@@ -5100,7 +5105,7 @@ definitions:
         type: string
       url:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportSource:
     required:
@@ -5111,5 +5116,22 @@ definitions:
         type: string
       target:
         additionalProperties: true
-    additionalProperties: false
+    additionalProperties: true
     type: object
+  ImportPackageRelationship:
+    required:
+      - parent
+      - child
+      - type
+    properties:
+      parent:
+        type: string
+      child:
+        type: string
+      type:
+        type: string
+      metadata:
+        additionalProperties: true
+        type: object
+    additionalProperties: true
+    type: object
diff --git a/anchore_engine/services/catalog/image_content/get_image_content.py b/anchore_engine/services/catalog/image_content/get_image_content.py
@@ -112,13 +112,13 @@ def hydrate_additional_data(self, image_content_data, image_report):
         if image_content_data.get("dockerfile", None):
             # Nothing to do here
             return helpers.make_image_content_response(
-                self.content_type, image_content_data
+                self.content_type, image_content_data[self.content_type]
             )
         try:
             if image_report.get("dockerfile_mode", None) != "Actual":
                 # Nothing to do here
                 return helpers.make_image_content_response(
-                    self.content_type, image_content_data
+                    self.content_type, image_content_data[self.content_type]
                 )
 
             for image_detail in image_report.get("image_detail", []):
@@ -149,5 +149,5 @@ def hydrate_additional_data(self, image_content_data, image_report):
                 )
             )
         return helpers.make_image_content_response(
-            self.content_type, image_content_data
+            self.content_type, image_content_data[self.content_type]
         )
diff --git a/anchore_engine/services/catalog/swagger/swagger.yaml b/anchore_engine/services/catalog/swagger/swagger.yaml
@@ -2299,6 +2299,10 @@ definitions:
         "$ref": "#/definitions/ImportDescriptor"
       schema:
         "$ref": "#/definitions/ImportSchema"
+      artifactRelationships:
+        items:
+          "$ref": "#/definitions/ImportPackageRelationship"
+        type: array
     additionalProperties: true
     type: object
   ImportDescriptor:
@@ -2310,7 +2314,7 @@ definitions:
         type: string
       version:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportDistribution:
     required:
@@ -2324,7 +2328,7 @@ definitions:
         type: string
       idLike:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportPackageLocation:
     required:
@@ -2334,7 +2338,7 @@ definitions:
         type: string
       layerID:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportPackage:
     required:
@@ -2347,6 +2351,8 @@ definitions:
       - cpes
       - metadataType
     properties:
+      id:
+        type: string
       name:
         type: string
       version:
@@ -2376,9 +2382,8 @@ definitions:
         type: string
       metadata:
         type: object
-    additionalProperties: false
+    additionalProperties: true
     type: object
-
   ImportSchema:
     required:
       - version
@@ -2388,7 +2393,7 @@ definitions:
         type: string
       url:
         type: string
-    additionalProperties: false
+    additionalProperties: true
     type: object
   ImportSource:
     required:
@@ -2399,6 +2404,22 @@ definitions:
         type: string
       target:
         additionalProperties: true
-    additionalProperties: false
+    additionalProperties: true
     type: object
-
+  ImportPackageRelationship:
+    required:
+      - parent
+      - child
+      - type
+    properties:
+      parent:
+        type: string
+      child:
+        type: string
+      type:
+        type: string
+      metadata:
+        additionalProperties: true
+        type: object
+    additionalProperties: true
+    type: object
diff --git a/anchore_engine/services/policy_engine/__init__.py b/anchore_engine/services/policy_engine/__init__.py
@@ -157,6 +157,7 @@ def _init_distro_mappings():
         DistroMapping(from_distro="rhel", to_distro="rhel", flavor="RHEL"),
         DistroMapping(from_distro="ubuntu", to_distro="ubuntu", flavor="DEB"),
         DistroMapping(from_distro="amzn", to_distro="amzn", flavor="RHEL"),
+        DistroMapping(from_distro="redhat", to_distro="rhel", flavor="RHEL"),
     ]
 
     # set up any data necessary at system init

diff --git a/anchore_engine/version.py b/anchore_engine/version.py
@@ -1,2 +1,3 @@
-version = "0.9.1"
+version = "0.9.2"
+
 db_version = "0.0.14"
diff --git a/anchore_manager/version.py b/anchore_manager/version.py
@@ -1 +1 @@
-version = "0.9.1"
+version = "0.9.2"