-fail-on-error doesn't work. reviewdog returns zero as exit code even if warnings are found in input

[suzuki-shunsuke]

This is a bug report of -fail-on-error option. Even if warnings are found in input, reviewdog command returns zero as exit code.

  -fail-on-error
    	Returns 1 as exit code if any errors/warnings found in input

How to reproduce

• Test reviewdog suzuki-shunsuke/test-github-action#146
• https://github.com/suzuki-shunsuke/test-github-action/actions/runs/4738975477/jobs/8413381617

$ reviewdog --version
0.14.1

$ tflint -v
TFLint version 0.46.0
+ ruleset.terraform (0.2.2-bundled)

main.tf

// hello

$ tflint --format checkstyle --module
<?xml version="1.0" encoding="UTF-8"?>
<checkstyle>
  <file name="">
    <error rule="terraform_required_version" line="0" column="0" severity="warning" message="terraform &#34;required_version&#34; attribute is required" link="https://github.com/terraform-linters/tflint-ruleset-terraform/blob/v0.2.2/docs/rules/terraform_required_version.md"></error>
  </file>
</checkstyle>

$ tflint --format checkstyle --module |
            reviewdog \
              -f checkstyle \
              -name tflint \
              -filter-mode nofilter \
              -reporter github-pr-check \
              -level warning \
              -fail-on-error 1
reviewdog: Reporting results for "tflint"
Warning: [tflint] reported by reviewdog 🐶
terraform "required_version" attribute is required

Raw Output:
:0:0: warning: terraform "required_version" attribute is required ()

$ echo "$?"
0

Expected behaviour

The exit code of reviewdog command is not zero because warning is outputted.

Actual behaviour

The exit code of reviewdog command is zero.

[dobesv]

I've been seeing the same thing, the output logs show errors but it doesn't exit 1. It doesn't even report the error to GitHub. It doesn't seem to be very consistent.

[fharper]

I had the same issue, so I decided to run vale manually. Here's my workaround steps:

- name: Get latest version of Vale
  id: lastversion
  uses: dvershinin/lastversion-action@v0.0.3
  with:
    repository: errata-ai/vale

- name: Install Vale
  run: |
    wget https://github.com/errata-ai/vale/releases/download/v${{ steps.lastversion.outputs.last_version }}/vale_${{ steps.lastversion.outputs.last_version }}_Linux_64-bit.tar.gz -O vale.tar.gz
    tar -xvzf vale.tar.gz vale
    rm vale.tar.gz

- name: Validate the syntax
  run: ./vale --config=.vale.ini *.md

The vale command has the proper exit code, so the workflow will fail or succeed normally.

You can skip the first step, and wget the direct link if you want to use a specific version, instead of using the latest.

[dobesv]

Here's an example of how one format reports failure, the other doesn't:

Run with eslint output format:

/path/to/file
  14:3  error  Delete `··`  prettier/prettier

✖ 1 problem (1 error, 0 warnings)
  1 error and 0 warnings potentially fixable with the `--fix` option.

  14:3  error  Delete `··`  prettier/prettier
reviewdog: input data has violations
ERROR: "lint:graphql" exited with 1.

Run with rdjson format:

{"source":{"name":"eslint","url":"https://eslint.org/"},"diagnostics":[{"message":"Delete `··`","location":{"path":"/home/dobes/projects/formative/app.fixups/packages/main/src/ui/client/teamMember/commandCenter/sections/ManageOneRosters/ManageOneRoster.graphql","range":{"start":{"line":14,"column":3},"end":{"line":14,"column":5}}},"severity":"ERROR","code":{"value":"prettier/prettier","url":"https://github.com/prettier/eslint-plugin-prettier#options"},"original_output":"{\"ruleId\":\"prettier/prettier\",\"severity\":2,\"message\":\"Delete `··`\",\"line\":14,\"column\":3,\"nodeType\":null,\"messageId\":\"delete\",\"endLine\":14,\"endColumn\":5,\"fix\":{\"range\":[220,222],\"text\":\"\"}}","suggestions":[{"range":{"start":{"line":14,"column":3},"end":{"line":14,"column":5}},"text":""}]}]}
ERROR: "lint:graphql" exited with 1.

[dobesv]

Actually, I think my comments above are inaccurate. I wasn't passing -format rdjson in those cases, so it wasn't parsing the linter output.

[ramonvermeulen]

Facing the same issue, using errata-ai/vale-action@reviewdog version 3.4.1 with reviewdog version 0.17.0.