Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print file name in output when tests fail #4457

Closed
anderseknert opened this issue Mar 22, 2022 · 2 comments · Fixed by #4461
Closed

Print file name in output when tests fail #4457

anderseknert opened this issue Mar 22, 2022 · 2 comments · Fixed by #4461

Comments

@anderseknert
Copy link
Member

anderseknert commented Mar 22, 2022

Sometimes you'll need to run tests over several directories, and in case one test fails, it's currently difficult to say exactly where without further grepping.

❯ find templates -type d -name 'opa' -exec opa test {} \;
data.policy.tests.test_deny_if_bucket_encryption_not_set: FAIL (292.875µs)
data.policy.tests.test_allow_if_public_access_blocked: FAIL (159.5µs)
--------------------------------------------------------------------------------
PASS: 1/3
FAIL: 2/3
PASS: 4/4
PASS: 3/3
PASS: 2/2

It would be useful if the default pretty --format printed the location of the file where failure(s) occured, aggregated per file (to avoid too much noise):

❯ find templates -type d -name 'opa' -exec opa test {} \;
/my/policies/policy_test1.rego:
data.policy.tests.test_deny_if_bucket_encryption_not_set: FAIL (292.875µs)
data.policy.tests.test_allow_if_public_access_blocked: FAIL (159.5µs)
--------------------------------------------------------------------------------
FAIL: 2/3
...
@tsandall
Copy link
Member

tsandall commented Mar 22, 2022

If we're printing the filename, is there much point in printing the virtual document path of the test? Personally I'd want something like:

/my/policies/policy_test1.rego:37: test_deny_if_bucket_encryption_not_set: FAIL (292.875µs)
/my/policies/policy_test1.rego:49: test_allow_if_public_access_blocked: FAIL (159.5µs)

This way I can click on the filepath/line number and jump to the test in question (vscode does this today).

@anderseknert
Copy link
Member Author

I suppose "pretty" is subjective :) My rationale for wanting to group this was that I think printing the location of each failing test will look messy for deeply nested directories, i.e:

/Users/anderseknert/git/opa/lab/my_test_directory/my/policies/policy_test1.rego:37: test_deny_if_bucket_encryption_not_set: FAIL (292.875µs)

I've found this quite common in CI/CD contexts, when some random long directory names have been generated for the run, though I guess one could avoid that by not providing the absolute path to opa test.

The "click to view" is a nifty feature though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants