Add assertion to check if the order of test paths is correct #856
Conversation
tests/cli_test_case.py
Outdated
| @@ -294,3 +295,15 @@ def tree_sorted(obj): | |||
| return obj | |||
|
|
|||
| self.assertEqual(tree_sorted(a), tree_sorted(b)) | |||
|
|
|||
| def assert_test_path_orderly_equal(self, a, b): | |||
There was a problem hiding this comment.
Let's define the contract of this method. What exactly does this method assert?
I ask because I was expecting a and b are symmetric, but L308-309 doesn't work that way. For exampe, if a is empty list, this assertion passes no matter what b is.
I'm almost tempted to just test a['events']['testPath']==b['events']['testPath']; Is that not what we want to do?
There was a problem hiding this comment.
I'm almost tempted to just test a['events']['testPath']==b['events']['testPath']; Is that not what we want to do?
I initially planed to do, but it's not possible because the value associated with the 'events' key is in random order.
There was a problem hiding this comment.
https://github.com/launchableinc/cli/pull/856/files#diff-b0c1be9f593a5643f1de3766e049337b1abde38f8fe1e6f244d09125bf8c03e0R191-R242 is an example to describe the situation. I'd like to check only the order of elements in the value of testPath key.
| @@ -20,27 +20,6 @@ def test_record_test_minitest(self): | |||
| self.assert_success(result) | |||
| self.assert_record_tests_payload('record_test_result.json') | |||
|
|
|||
| # For testing https://github.com/launchableinc/cli/pull/846/files#r1591707246. | |||
There was a problem hiding this comment.
Yes, it is. I've added this test for testing the order as a temporary solution. However, by adding this assertion, this test method will be meaningless.
|
|
Thanks! There's some loose ends when duplicates are present but now the intent is clar, so that's OK for now. |
Currently, we do not verify the order of JSON at https://github.com/launchableinc/cli/blob/main/tests/cli_test_case.py#L221 because we evaluate JSON in an order-agnostic manner. However, we aim to ensure the order of test paths is accurate.
assert_json_orderless_equaldisregards the order of items intentionally. Consequently, I have introduced theassert_test_path_orderly_equalmethod to address this requirement, instead of modifyingassert_json_orderless_equal.@kohsuke
I think you designed these methods in #91. Can you take a look at this PR?