fail-on-error=false (or left on default) still exits 1 using Github checks

[ohookins]

I may have misunderstood the code, so if I have then my apologies. My expectation for -fail-on-error=false or leaving the option out entirely (which should default to false) is that reviewdog will error with return code 0 in all situations except if an internal error or misconfiguration occurs. This is clearly not happening:

reviewdog: Reporting results for "rspec"
  reviewdog: found at least one result in diff
  Error: [rspec] reported by reviewdog 🐶
<redacted test case output>

  Raw Output:
<redacted test case output>
Error: Process completed with exit code 1.

This doubles the number of test fail annotations we have on every Github Actions run, as well as making the overall effect much harder to understand since it is more noisy.

It seems that the option to fail on error is properly handled for Github Actions runs in postResultSet:

reviewdog/cmd/reviewdog/doghouse.go

Lines 172 to 174 in 206c018

	if opt.failOnError && (res.Conclusion == "failure") {
	return fmt.Errorf("[%s] Check conclusion is %q", name, res.Conclusion)
	}

And yet in reportResults the logic appears to mandate exiting non-zero if we have anything that is at the error level:

reviewdog/cmd/reviewdog/doghouse.go

Lines 227 to 230 in 206c018

	// If it's not running in GitHub Actions, reviewdog should exit with 1
	// if there are at least one result in diff regardless of error level.
	shouldFail = shouldFail || !cienv.IsInGitHubAction() ||
	!(results.Level == "warning" || results.Level == "info")

Perhaps there's an error in the logic? I'm running it in Github Actions, so logic short-circuiting is not active and passes through to the comparison to the result.Level. If the result.Level is error, the last section of this statement evaluates to true and we end up with an error. Indeed, if it is not running in Github Actions it seems like it will exit non-zero irrespective of the result.Level which doesn't match the comment's intention. Unless I'm misreading this?

This issue may also be related to the following, although I'm not positive:

• Better error handling behavoir (--fail-on-error) #856
• -fail-on-error flag causes unexpected behavior #989
• Failed linter on GitHub Actions run does not cause check to fail when fail_on_error is true #949