CLI for goconvey v1.6.6 "loses" test results

[mt35-rs]

I've created a repository to reproduce what I'm seeing. But I suspect anybody using the v1.6.6 CLI is going to run into this (perhaps enough people simply haven't upgraded yet?). Although it is possible this is somehow specific to me and my setup.

The repository can be found here and the README describes the issue but I'll repeat the essential details here:

When you run goconvey v1.6.6 on the source in that repository, you'll see a bunch of lines that look
like this in the output:

2021/10/20 16:04:11 goconvey.go:61: Initial configuration: [host: 127.0.0.1] [port: 8080] [poll: 250ms] [cover: true]
2021/10/20 16:04:11 tester.go:19: Now configured to test 10 packages concurrently.
2021/10/20 16:04:11 goconvey.go:203: Serving HTTP at: http://127.0.0.1:8080
2021/10/20 16:04:11 goconvey.go:105: Launching browser on 127.0.0.1:8080
2021/10/20 16:04:11 integration.go:122: File system state modified, publishing current folders... 0 4904277887
2021/10/20 16:04:11 goconvey.go:118: Received request from watcher to execute tests...
2021/10/20 16:04:12 executor.go:69: Executor status: 'executing'
2021/10/20 16:04:12 coordinator.go:46: Executing concurrent tests: github.com/mtiller/gct
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: .x
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: Failures:
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line:
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: * /home/mt35/Source/goconvey-test/add_test.go
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: Line 12:
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: Expected: '6'
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: Actual:   '5'
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: (Should be equal)
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line:
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line:
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line: 2 total assertions
Potential error parsing output of github.com/mtiller/gct ; couldn't handle this stray line:
2021/10/20 16:04:12 parser.go:24: [failed]: github.com/mtiller/gct
2021/10/20 16:04:12 executor.go:69: Executor status: 'idle'
2021/10/20 16:04:13 goconvey.go:113:

But when you run v1.6.4, you don't see this:

2021/10/20 16:03:32 goconvey.go:61: Initial configuration: [host: 127.0.0.1] [port: 8080] [poll: 250ms] [cover: true]
2021/10/20 16:03:32 tester.go:19: Now configured to test 10 packages concurrently.
2021/10/20 16:03:32 goconvey.go:178: Serving HTTP at: http://127.0.0.1:8080
2021/10/20 16:03:32 integration.go:122: File system state modified, publishing current folders... 0 4904277887
2021/10/20 16:03:32 goconvey.go:105: Launching browser on 127.0.0.1:8080
2021/10/20 16:03:32 goconvey.go:118: Received request from watcher to execute tests...
2021/10/20 16:03:32 executor.go:69: Executor status: 'executing'
2021/10/20 16:03:32 coordinator.go:46: Executing concurrent tests: github.com/mtiller/gct
2021/10/20 16:03:33 parser.go:24: [failed]: github.com/mtiller/gct
2021/10/20 16:03:33 executor.go:69: Executor status: 'idle'
2021/10/20 16:03:33 goconvey.go:113:

This might seem like a superficial issue, but it is indicative of a deeper
problem which manifests itself more seriously in the web interface.
Specifically, the test failure in the code triggers some issue with parsing
output in the goconvey CLI tool which causes the all results of that test to
be complete masked (successes and failures).

The README in the repository
includes some figures as well to see what I mean. I should add that those figures
don't show accurate coverage information. But that is a separate issue that a)
already has been reported and b) has an easy work around. So don't worry about
coverage. The issue here is the non-reporting of results from any test where
an assertion fails.

My current workaround is to downgrade the CLI to v1.6.4 (I can use either v1.6.4 or v1.6.6
in my go.mod file to import Convey and So that doesn't have any impact, FYI).

I really like the goconvey approach and I used to use it in a previous life when I was using
Go (under my @xogeny persona). So let me know if there is anything else I can do to
help fix this.

[mt35-rs]

I should add that I'm running this under Ubuntu 20.04 (under WSL2).

[riannucci]

Thanks for taking the time to dig into this and file a solid issue!

Definitely seems like a regression... skimming the diff between 1.6.4 and 1.6.6, I wonder if maybe https://github.com/smartystreets/goconvey/compare/v1.6.4..v1.6.6#diff-ae6722696d6e9ca440dc7ee0ce295e73df27151658a4dbdd422445a2ddf26636R53 is related?

[riannucci]

Hm... I'm not able to repro this with a clone of your repo and goconvey 1.6.6 (using the same version of the goconvey binary as the test library... can you confirm that you don't have a mismatch by accident?)

[riannucci]

Nevermind! I, in fact, had the wrong binary version. Can definitely repro this.

[mt35-rs]

Just to be clear, things work correctly so long as there is no error being reported. I don't know enough about how goconvey does this parsing, so I'm not much help as to what the cause is it. It certainly seems plausible that the additional handling of Gingko test output perhaps deranged the existing handling.

I'm surprised more people haven't noticed this issue yet (I guess they all only write tests that pass ;-)). But I suspect people will start coming out of the woodwork as the latest changes start to percolate out. The fact that we can use the v1.6.6 library with the v1.6.4 CLI suggests to me that people simple haven't updated the CLI yet. Hopefully this can be addressed to avoid a slew of concerns being raised once people do start updating to newer versions of the CLI.

[riannucci]

So I think this actually caused by https://github.com/smartystreets/goconvey/compare/v1.6.4..v1.6.6#diff-4f4038338de6c15311b73ae5c1f09ab0be69e49a9a0f1c0d5c5408a53cb5e682R6.

I should have a fix for this (and spin a new version) today.

[riannucci]

Bah, pushed this the wrong way

[riannucci]

Ok, now fixed and included in v1.6.7

[mt35-rs]

Excellent. Thanks for the quick turn around!