Improve test result accessibility #3082

gibson042 · 2022-08-06T19:28:30Z

Explain diff gutter symbols.
Add test result labels (color covers figure and label, and indicates passed-as-expected vs. not).
Bold the source line of errors in code excerpts.

Color covers figure and label, and indicates passed-as-expected vs. not (expected/unexpected fail or unexpected pass). Fixes avajs#2919

gibson042 · 2022-08-06T19:42:33Z

This produces lots of failures like the following; how do I update the [TAP] test expectations in bulk?

        not ok 43 - should be equal
          ---
          compare: ===
          at:
            line: 59
            column: 6
            file: file:///home/runner/work/ava/ava/test-tap/assert.js
            function: assertFailure
          stack: |
            assertFailure (file://test-tap/assert.js:59:6)
            failsWith (file://test-tap/assert.js:98:2)
            Test.<anonymous> (file://test-tap/assert.js:277:2)
            file://test-tap/assert.js:202:1
          diff: |
            --- expected
            +++ actual
            @@ -1,1 +1,1 @@
            -Difference:
            +Difference (- actual, + expected):
          ...

gibson042 · 2022-08-07T06:55:30Z

Found it: https://github.com/avajs/ava/blob/main/test-tap/reporters/readme.md

This _seems_ to work, but the resulting interaction between CommonJS and ESM feels a bit Zalgo-y. It may be better for modules to keep setting global options and chalk options in parallel while the former remains CommonJS, but I wanted to see what simplification would look like.

gibson042 · 2022-08-07T20:05:09Z

lib/worker/options.cjs

+		if (setChalk) {
+			setChalk(options.chalkOptions);
+		} else {
+			chalk.then(chalk => chalk.set(options.chalkOptions));
+		}


This is from 1b64bff, an independent experiment that can be reverted if deemed unsuccessful.

This seems to work, but the resulting interaction between CommonJS and ESM
feels a bit Zalgo-y. It may be better for modules to keep setting global
options and chalk options in parallel while the former remains CommonJS,
but I wanted to see what simplification would look like.

Is this needed for this PR? Can we discuss it in a separate PR? It's all a little complicated I know, do we need to change it? What do you think the benefits are?

I'd prefer to keep it, because otherwise the new tests don't correspond with reality (because worker output does not consistently respect color settings) and will undergo a fair amount of churn. But if you feel strongly that it should be separated, I can do that.

I had a go at removing it, and other than test-tap/assert.js requiring an extra stripAnsi() there's no impact. I think I prefer the "set once" approach rather than having code react to changes.

lib/cli.js

lib/concordance-options.js

novemberborn · 2022-08-13T13:50:36Z

lib/worker/options.cjs

+		if (setChalk) {
+			setChalk(options.chalkOptions);
+		} else {
+			chalk.then(chalk => chalk.set(options.chalkOptions));
+		}


Is this needed for this PR? Can we discuss it in a separate PR? It's all a little complicated I know, do we need to change it? What do you think the benefits are?

novemberborn · 2022-08-13T13:51:00Z

@sindresorhus what do you think?

novemberborn · 2022-08-13T13:51:41Z

Thank you @gibson042, very exciting!

Note that we're seeing similar CI failures on the main branch, so let's not worry about those here.

sindresorhus · 2022-08-15T00:13:28Z

I agree with almost everything. However, I'm not a big fan of the PASS: and FAIL: noise. I think the icons are clear enough for the common case and we could instead only use the prefixes for the other cases like EXPECTED FAIL and SKIP.

gibson042 · 2022-08-16T16:52:17Z

I agree with almost everything. However, I'm not a big fan of the PASS: and FAIL: noise. I think the icons are clear enough for the common case and we could instead only use the prefixes for the other cases like EXPECTED FAIL and SKIP.

@sindresorhus Without "PASS" and "FAIL", vision-impaired consumers are left to infer semantics from information like "check mark" and "multiplication symbol" (the latter improved a bit after sindresorhus/figures#92 ). I'd prefer to keep the verbosity in support of accessibility and output consistency, but if you still disagree then let me know and I'll remove them.

novemberborn · 2022-08-22T08:39:04Z

@gibson042 I should have some time this week to help get this over the line.

…ult-accessibility

novemberborn · 2022-08-27T16:13:12Z

I agree with almost everything. However, I'm not a big fan of the PASS: and FAIL: noise. I think the icons are clear enough for the common case and we could instead only use the prefixes for the other cases like EXPECTED FAIL and SKIP.

@sindresorhus Without "PASS" and "FAIL", vision-impaired consumers are left to infer semantics from information like "check mark" and "multiplication symbol" (the latter improved a bit after sindresorhus/figures#92 ). I'd prefer to keep the verbosity in support of accessibility and output consistency, but if you still disagree then let me know and I'll remove them.

@sindresorhus how strongly do you feel about this?

I do like the consistency.

sindresorhus · 2022-08-28T16:19:08Z

@sindresorhus how strongly do you feel about this?

I don't feel strongly about it.

sindresorhus · 2022-08-28T16:21:00Z

I do like the consistency.

Most assertion will be pass or fail, so there wouldn't be much inconsistency in practice. And it may even have been beneficial to make the uncommon cases more obvious. The result now is also more noisy.

Use lowercase, since AVA doesn't output anything in all caps. Wrap in square brackets to differentiate from the test title. Remove prefix for passed tests. We assume that most tests pass most of the time.

To better differentiate from the test title.

novemberborn · 2022-09-04T14:20:29Z

Project name aside, we don't really use all caps. I've pushed a commit to use lowercase and put it in square brackets.

Most of the time, most tests pass. So I've removed the prefix for passed tests.

I also noticed that the assertion message was only differentiated from the test title by color, so I've made that italic.

gibson042 added 3 commits August 6, 2022 14:25

Explain diff gutter symbols

6e1dbb9

Fixes avajs#1558

Add test result labels

1c92630

Color covers figure and label, and indicates passed-as-expected vs. not (expected/unexpected fail or unexpected pass). Fixes avajs#2919

Bold the source line of errors in code excerpts

93070fe

gibson042 mentioned this pull request Aug 6, 2022

Main cross character is incorrect sindresorhus/figures#92

Closed

gibson042 added 4 commits August 6, 2022 20:40

DRY out writeTestSummary

ef19867

Better propagate color configuration

017a45a

TAP reporter: Strip ANSI control sequences from error labels

49fbe24

Update tests

f1beca2

gibson042 commented Aug 7, 2022

View reviewed changes

novemberborn reviewed Aug 13, 2022

View reviewed changes

novemberborn added 3 commits August 26, 2022 17:03

Merge remote-tracking branch 'origin/main' into avajsgh-2919-test-res…

36d764c

…ult-accessibility

Undo changes to lib/chalk.js

9b45dc3

Keep relying on Chalk to parse --color argv

fcc1f07

novemberborn added 2 commits September 4, 2022 16:13

Tweak result prefixes

0c0fd21

Use lowercase, since AVA doesn't output anything in all caps. Wrap in square brackets to differentiate from the test title. Remove prefix for passed tests. We assume that most tests pass most of the time.

Print assertion error in italics

ee9f000

To better differentiate from the test title.

Update snapshot

2449d74

novemberborn merged commit 797287a into avajs:main Sep 4, 2022

gibson042 mentioned this pull request Jan 25, 2023

fix: add ava to junit parser Agoric/agoric-sdk#6847

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve test result accessibility #3082

Improve test result accessibility #3082

gibson042 commented Aug 6, 2022

gibson042 commented Aug 6, 2022

gibson042 commented Aug 7, 2022

gibson042 Aug 7, 2022

novemberborn Aug 13, 2022

gibson042 Aug 16, 2022

novemberborn Aug 27, 2022

novemberborn Aug 13, 2022

novemberborn commented Aug 13, 2022

novemberborn commented Aug 13, 2022

sindresorhus commented Aug 15, 2022

gibson042 commented Aug 16, 2022

novemberborn commented Aug 22, 2022

novemberborn commented Aug 27, 2022

sindresorhus commented Aug 28, 2022

sindresorhus commented Aug 28, 2022

novemberborn commented Sep 4, 2022

Improve test result accessibility #3082

Improve test result accessibility #3082

Conversation

gibson042 commented Aug 6, 2022

gibson042 commented Aug 6, 2022

gibson042 commented Aug 7, 2022

gibson042 Aug 7, 2022

Choose a reason for hiding this comment

novemberborn Aug 13, 2022

Choose a reason for hiding this comment

gibson042 Aug 16, 2022

Choose a reason for hiding this comment

novemberborn Aug 27, 2022

Choose a reason for hiding this comment

novemberborn Aug 13, 2022

Choose a reason for hiding this comment

novemberborn commented Aug 13, 2022

novemberborn commented Aug 13, 2022

sindresorhus commented Aug 15, 2022

gibson042 commented Aug 16, 2022

novemberborn commented Aug 22, 2022

novemberborn commented Aug 27, 2022

sindresorhus commented Aug 28, 2022

sindresorhus commented Aug 28, 2022

novemberborn commented Sep 4, 2022