Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue-6559 Add ability to create decision labels within a policy #6681

Open
wants to merge 114 commits into
base: main
Choose a base branch
from

Conversation

tsidebottom
Copy link

Why the changes in this PR are needed?

Adds a DecisionLabel Field (a Map Object) to the Decision Log output to track Policy Result Data.

What are the changes in this PR?

  • A new field (DecisionLabel) was added to the Decision Log
  • This field was propagated backwards through the Policy Evaluation process
    • includes updates to files in Topdown, Rego, and SDK domains
  • A new builtin (DecisionLabelAdd) was created to populate the new field during evaluation

Notes to assist PR review:

Further comments:

Addresses #6559

Copy link

netlify bot commented Apr 5, 2024

Deploy Preview for openpolicyagent ready!

Name Link
🔨 Latest commit 4070f53
🔍 Latest deploy log https://app.netlify.com/sites/openpolicyagent/deploys/6621259b68b7f400085fb353
😎 Deploy Preview https://deploy-preview-6681--openpolicyagent.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@ashutosh-narkar
Copy link
Member

Thanks for the contribution @tsidebottom. FYI we've added some comments on #6559 about the proposed feature including the latest one here. It would be helpful to identify alternate ways of achieving what's needed for #6559. Thanks.

ashutosh-narkar and others added 28 commits April 17, 2024 13:52
Currently the `raise_error` flag is not honored during the
input validation step. So `http.send` will return an error if
input validation fails irrespective of the `raise_error` flag
status. This change attempts to fix that.
Also the description of the `raise_error` flag is updated to
reflect actual behavior.

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
DecisionLabel will be populated by the Custom Built-in.

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Actual definition and code will go here.

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Added to the EvalContext Struct.

Created a method to pull the DecisionLabel object in from the SDK (opa.go; that update is coming shortly).

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
yarivg and others added 24 commits April 17, 2024 13:52
Signed-off-by: yarivg <yarivgavriel2@gmail.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
These integrations feature on the learning rego page and the terraform
pages respectively.

Signed-off-by: Charlie Egan <charlie@styra.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
The cost of this can become non-trivial with larger inputs.

Signed-off-by: Teemu Koponen <koponen@styra.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
This commit adds the possibility to configure the plugin manager with
custom options.

It will allow SDK users to override the options already provided by the
SDK and to futher customize it with configurations that were not
previously available. This is an advanced feature as it requires
some knowledge about the inner workings of OPA.

One use case for this is to provide a prometheus registerer and have the
status plugin metrics available for the client to use it in a
/metrics endpoint, for example.

resolves open-policy-agent#6662

Signed-off-by: Francisco Rodrigues <ednofco@gmail.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
This commit fixes a panic that could occur when `opa build` was provided
an entrypoint from both a CLI flag, and via entrypoint metadata
annotation.

The fix is simple: deduplicate the slice of entrypoint refs that the
compiler uses, before compiling WASM or Plan targets.

Fixes: open-policy-agent#6661

Co-authored-by: Daniel Herzig <danielherzig96@gmail.com>
Signed-off-by: Philip Conrad <philipaconrad@gmail.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
This time for v0QueryPath, v1DataGet, and v1DataPost.

Signed-off-by: Teemu Koponen <koponen@styra.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
https://go.dev/doc/devel/release#go1.22.2

Signed-off-by: Stephan Renatus <stephan@styra.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
…-agent#6671)

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.22.0 to 0.23.0.
- [Commits](golang/net@v0.22.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
…policy-agent#6673)

Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.62.1 to 1.63.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.62.1...v1.63.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model) from 0.5.0 to 0.6.1.
- [Release notes](https://github.com/prometheus/client_model/releases)
- [Commits](prometheus/client_model@v0.5.0...v0.6.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_model
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
In workflow runs like this:
https://github.com/open-policy-agent/opa/actions/runs/7803493290/job/21283458848#step:3:317

We can see two problems. This commit is meant to address them.

First, the test failed with this message:

```
expected unknown certificate authority error but got: Get "https://127.0.0.1:38699/v1/data": write tcp 127.0.0.1:52786->127.0.0.1:38699: write: connection reset by peer
```

Now this step in the test is retried like the other steps in the test
since it can fail too.

Second, the error `failed to reload TLS config` appears many times in
the logs for that test. This issue is caused by the server attempting to
read the new cert, key, and CA contents from disk while they are still
being written to. This PR also introduces a 100ms pause between upto 5
attempts to reload the config for any given change to the state on disk.
This should mean that the error is seen only when is is actually an
issue and the reload has failed after a reasonable time. In most cases,
running locally, the reload happens without error on the first run.

Signed-off-by: Charlie Egan <charlie@styra.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
…ty-printed" and/or line-prefixed JSON (open-policy-agent#6636)

Fixes open-policy-agent#6630

Signed-off-by: Sean Williams <72675818+sean-r-williams@users.noreply.github.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
This reflects the reality -- we hadn't been sure why the dependabot update had
not increased the stanza when it should have; but doing so now should unbreak
the nightly tests.

Signed-off-by: Stephan Renatus <stephan@styra.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
…-agent#6680)

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.7.14 to 1.7.15.
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](containerd/containerd@v1.7.14...v1.7.15)

---
updated-dependencies:
- dependency-name: github.com/containerd/containerd
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.63.0 to 1.63.2.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.63.0...v1.63.2)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Improving memory footprint and execution time of deps command for policies with high dependency connectivity.

Fixes: open-policy-agent#6685
Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: kunal.das <kd.kunaldas92@gmail.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Adding a global `rego_version` attribute to bundle manifest, to inform OPA runtime about what rego-version (v0/v1) to use to parse/compile contained Rego files.
The rego-version of individual Rego files can be overridden through the `file_rego_versions` manifest attribute.

Implements: open-policy-agent#6578

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
…policy-agent#6689)

Fixing issue where active parser options aren't propagated to module reload during bundle activation.

Signed-off-by: Francisco Rodrigues <ednofco@gmail.com>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Documenting bundle rego-version.

Signed-off-by: Johan Fylling <johan.dev@fylling.se>
Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
The DecisionLabel is no longer a Field of the DecisionOptions Struct because the field will always be nil. It is now generated directly within the Decision() Function where it can be populated. Due to this, the scenario being tested is impossible. For coverage, the scenario being tested by TestDecisionLoggingWithDecisionLabel is formally covered by the tests which confirm the proper function of the DecisionLabelAdd Builtin Function.

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
tsidebottom and others added 5 commits April 17, 2024 13:59
Signed-off-by: Thomas Sidebottom <91754088+tsidebottom@users.noreply.github.com>
Using `goimports` to format the imports

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
The DecisionLabelAdd Builtin was added to the capabilities.json file to allow it to be tracked.

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
I missed this even after the GitHub Check told me to look here...

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
I added it to the list, but I forgot to add the actual description of the Builtin. Using the Check Generated Job version.

Signed-off-by: Thomas Sidebottom <thomas.sidebottom@va.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet