-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inconsistent directory requirements for --output
vs. --gather-test-outputs-in
flag
#594
Comments
That heredoc problem is known, see #277. I need to look into why the Flags handle existing directories differently but I would assume its just an oversight. |
My bad. I uncovered that totally on accident (and only due to creating a self-contained reproduction). It's unlikely I would have encountered it during real-world usage.
As you'll discover, there is one "obvious" easy solution to make those two consistent with each other. But it was my lack of usage of all the other flags (that use directories), and not knowing if there was an underlying philosophy, that stopped me from submitting the obvious/easy two-character PR. While it was probably implied, I should have explicitly added that this is not at all a blocker. It took me a while to figure out the issue and solution (for reasons unrelated to bats itself), but both flags are fully usable once you know the trick. |
After having looked into the issue I am more certain that this is not an oversight. The rationale behind requiring the folder not to exist is that you don't get mixed up outputs from a previous run with the current one. This problem is mainly due to the fact that a report is a single file which can be replaced. While the outputs are one file per test, so you might get only partial overwrites. This issue might be solved by better documentation or by a warning for a non empty directory. |
That rationale makes perfect sense in the context of allowing reports to be be written to a populated directory and refusing to write test outputs to a populated directory. But I would think an empty directory would be effectively "the same" in regards to the risks you mentioned...? So right now my script has
Maybe in that context you'll see why it's a little awkward. (Or maybe not.) But yes, even if you decide the behavior should remain as-is, calling out the existence/non-existence requirements in the docs (and/or warnings/diagnostics to the errors) should help anyone else that gets confused. Thanks! |
You are welcome to create a PR for the empty directory check. |
This addresses the usability from bats-core#594. The rationale for the previous behavior (not allowing the directory to exist) makes sense; test executions to the same to the same location, over time, aren't guaranteed to generate the same output files (in name or quantity), leading to older test outputs bleeding together with newer ones. That reasoning is still maintained.
This addresses the usability from bats-core#594. The rationale for the previous behavior (not allowing the directory to exist) makes sense; test executions to the same to the same location, over time, aren't guaranteed to generate the same output files (in name or quantity), leading to older test outputs bleeding together with newer ones. That reasoning is still maintained.
Fixed in #603 |
Describe the bug
Disclaimer: I'm choosing to file this as "bug" because it (subjectively) fails "principle of least surprise" and I was unable to find documentation to explain (or hint at) the behavior.
I am simply trying to capture both test results and the outputs together. My desire was to capture both in sibling directories (i.e., a common parent). With no (obvious) documentation stating otherwise, I guessed/assumed creating the common parent directory would be sufficient and the two children would be automatically created. That resulted in an error. I then proceeded to pre-create both children directories. That also resulted in an error.
To Reproduce
Steps to reproduce the behavior:
I thought it might be clearer to assert the undesirable behavior and allow the test names to tell the story. So in this case, a passing test is actually a negative thing 😄 .
Contents of
wut.bats
:Incidentally, this actually shows a second pseudo-bug: using the
@test
marker in a heredoc still gets picked up as a test. Thesed
hack was to avoid getting an "expected X tests but ran Y" warning. I'm assuming this is an extremely atypical pattern, so didn't think it warranted filing an issue. But I'd be happy to file a report if you feel it should be addressed.Result (current and previous version):
Expected behavior
I would expect one of the following:
Environment (please complete the following information):
5.1.16
(version present in the bats docker image)Additional context
I am relatively new to this tool and have yet to explore all its features. Maybe, with extended use, it would be obvious why it behaves the way it does. If I was confident that it was a bug and that there was a "correct' answer, I likely would have contributed a PR alongside this report. But I'm not, so I didn't.
The text was updated successfully, but these errors were encountered: