New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(report): Apply ignore policies from a directory #6338
Open
dstrelbytskyi
wants to merge
7
commits into
aquasecurity:main
Choose a base branch
from
datarobot:ignore-policy-dir
base: main
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
a7c071f
Ignore policy from a directory
dstrelbytskyi 0847f6f
Added tests covering ignore policies in directory
dstrelbytskyi 8186412
Update docs
dstrelbytskyi d395694
Removed unwanted logging. Improved test.
dstrelbytskyi 92a3efb
Merge branch 'main' into ignore-policy-dir
dstrelbytskyi c8231bd
Final test fix
dstrelbytskyi cc214bb
Refactoring of the function finding policy files
dstrelbytskyi File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -3,17 +3,20 @@ package result | |||||
import ( | ||||||
"context" | ||||||
"fmt" | ||||||
"io/fs" | ||||||
"os" | ||||||
"path/filepath" | ||||||
"sort" | ||||||
|
||||||
"github.com/open-policy-agent/opa/bundle" | ||||||
"github.com/open-policy-agent/opa/rego" | ||||||
"github.com/samber/lo" | ||||||
"golang.org/x/exp/maps" | ||||||
"golang.org/x/exp/slices" | ||||||
"golang.org/x/xerrors" | ||||||
|
||||||
dbTypes "github.com/aquasecurity/trivy-db/pkg/types" | ||||||
"github.com/aquasecurity/trivy/pkg/log" | ||||||
"github.com/aquasecurity/trivy/pkg/sbom/core" | ||||||
sbomio "github.com/aquasecurity/trivy/pkg/sbom/io" | ||||||
"github.com/aquasecurity/trivy/pkg/types" | ||||||
|
@@ -69,8 +72,16 @@ func FilterResult(ctx context.Context, result *types.Result, ignoreConf IgnoreCo | |||||
filterLicenses(result, severities, opt.IgnoreLicenses, ignoreConf) | ||||||
|
||||||
if opt.PolicyFile != "" { | ||||||
if err := applyPolicy(ctx, result, opt.PolicyFile); err != nil { | ||||||
return xerrors.Errorf("failed to apply the policy: %w", err) | ||||||
// Get ignore policy files from the input path (either file or files in dir) | ||||||
policyFiles, err := findPolicyFiles(opt.PolicyFile) | ||||||
if err != nil { | ||||||
return err | ||||||
} | ||||||
|
||||||
for _, policyFile := range policyFiles { | ||||||
if err := applyPolicy(ctx, result, policyFile); err != nil { | ||||||
return xerrors.Errorf("failed to apply ignore policy %s: %w", policyFile, err) | ||||||
} | ||||||
} | ||||||
} | ||||||
sort.Sort(types.BySeverity(result.Vulnerabilities)) | ||||||
|
@@ -239,6 +250,42 @@ func summarize(status types.MisconfStatus, summary *types.MisconfSummary) { | |||||
} | ||||||
} | ||||||
|
||||||
func findPolicyFiles(policiesPath string) ([]string, error) { | ||||||
fi, err := os.Stat(policiesPath) | ||||||
if err != nil { | ||||||
return nil, xerrors.Errorf("failed to analyze ignore policy path %q: %w", policiesPath, err) | ||||||
} | ||||||
|
||||||
// The ignore policy option is a file | ||||||
if !fi.IsDir() { | ||||||
return []string{ | ||||||
policiesPath, | ||||||
}, nil | ||||||
} | ||||||
|
||||||
// If the ignore policy option is a dir find rego files in it | ||||||
var files []string | ||||||
if err = filepath.WalkDir(policiesPath, func(path string, d fs.DirEntry, err error) error { | ||||||
if err != nil { | ||||||
return err | ||||||
} | ||||||
if !d.Type().IsRegular() || filepath.Ext(path) != bundle.RegoExt { | ||||||
return nil | ||||||
} | ||||||
|
||||||
files = append(files, path) | ||||||
return nil | ||||||
}); err != nil { | ||||||
return nil, xerrors.Errorf("failed to find policy files in %q: %w", policiesPath, err) | ||||||
} | ||||||
|
||||||
if len(files) == 0 { | ||||||
log.Logger.Warnf("No ignore policies found in %q", policiesPath) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
} | ||||||
|
||||||
return files, nil | ||||||
} | ||||||
|
||||||
func applyPolicy(ctx context.Context, result *types.Result, policyFile string) error { | ||||||
policy, err := os.ReadFile(policyFile) | ||||||
if err != nil { | ||||||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
package trivy | ||
|
||
import data.lib.trivy | ||
|
||
default ignore=false | ||
|
||
ignore { | ||
input.AVDID != "AVD-ID100" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
package trivy | ||
|
||
import data.lib.trivy | ||
|
||
default ignore=false | ||
|
||
ignore { | ||
input.VulnerabilityID != "CVE-2019-0001" | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMHO, loading all Rego files together and evaluating the policies collectively is more intuitive for Rego users than loading and repeatedly evaluating individual Rego files. The current implementation may not work correctly if variable definitions are spread across multiple files.
To address this, using the
rego.Load()
function to load all Rego files from a directory recursively would be more appropriate. This ensures that all files are considered together, allowing for proper resolution of variables, rules, and dependencies.However, since I've been away from OPA recently, I would greatly appreciate insights from @simar7 to ensure the correctness and effectiveness of this approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 I'd also like to mention that running
.PrepareForEval()
(and the eventual evaluation itself) is an expensive operation. Doing it recursively on a directory that is user defined to load all files that match the Rego extension, whether relevant or not, can be costly.However if we go down the route of loading all rego files via
rego.Load()
as @knqyf263 mentioned, I'm not sure if we can have multiple checks that can contain the same default as the compiler will error out. Since Rego checks often contain defaults that result in "fail-close" type of checks, this would be a common occurrence (e.g. multiple checks havingdefault allow=false
).Maybe a safer route is to allow this flag to have values that can be a list of filenames rather than a directory? This would limit the scope of rego files that get loaded and evaluated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, we need to understand the use case precisely. @dstrelbytskyi Could you elaborate on it?