Add `expect-error` input to force PR check green on expected failure #1177

angelapwen · 2022-08-12T09:12:55Z

In #1159 we introduced a PR check to confirm that when the analyze workflow fails, partial debug artifacts are still uploaded. This caused 2 PR checks that always failed with a red X, which was not ideal for user experience.

This change adds an expect-error optional input to the analyze workflow, which is set by default to false. Only in the PR Check that intentionally causes a failure in analyze is it set to true (though it could also be used in future cases where we expect a failing job). If this parameter is true, we do not call core.setFailed(...) on the job so it succeeds.

If expect-error is set in any run not created by this repo, or a fork of it, we throw an error.

If expect-error is set to true, but no error in the run was found, the job fails.

Merge / deployment checklist

Confirm this change is backwards compatible with existing workflows.
Confirm the readme has been updated if necessary.
Confirm the changelog has been updated if necessary.

angelapwen · 2022-08-12T11:15:04Z

analyze/action.yml

@@ -66,6 +66,10 @@ inputs:
    default: ${{ github.token }}
  matrix:
    default: ${{ toJson(matrix) }}
+  expect-error:
+    description: For PR checks. If true, the Action will not set Actions status to failure.


May need to make this more descriptive so that our users don't set this to true when they don't mean to? I don't think I see a hidden option for input parameters.

How about this?

Suggested change

description: For PR checks. If true, the Action will not set Actions status to failure.

description: [Internal] If true, the Action will not set Actions status to failure.

Hm, this caused some failures — I wonder if the bracket symbols aren't able to be parsed correctly.

I think adding double quotes around it resolved it.

Consider throwing an error inside the code if this is set but the repo is not github/codeql-action or a fork. (You should be able to get that information from the Actions event context.)

I've added this code — still doing a final test of:

that if it's set in another repo, the error throws

that if it's set in this repo, the test passes

that if it's set in a fork of this repo, the test passes

that if it's set in this repo but no error is found, an error indicating this is thrown

aeisenberg · 2022-08-12T14:25:57Z

analyze/action.yml

@@ -66,6 +66,10 @@ inputs:
    default: ${{ github.token }}
  matrix:
    default: ${{ toJson(matrix) }}
+  expect-error:
+    description: For PR checks. If true, the Action will not set Actions status to failure.


How about this?

Suggested change

description: For PR checks. If true, the Action will not set Actions status to failure.

description: [Internal] If true, the Action will not set Actions status to failure.

aeisenberg · 2022-08-12T14:31:40Z

src/analyze-action.ts

@@ -183,7 +183,10 @@ async function run() {
  } catch (origError) {


Note that if an error isn't thrown, we need to set the job to a failure if this input is set to true.

It's a little more complicated than that since I see there are two places where core.setFailed is being called. Do we need to call both of them? It looks like the one in runWrapper is higher up the call chain. Maybe we don't need the one in run.

Note that if an error isn't thrown, we need to set the job to a failure if this input is set to true.

I'm a little confused: if an error isn't thrown, then why do we need to set the job to a failure? In that case, the job should succeed and we want that, right? Or do you mean that if we expect-error that there must be an error, otherwise there was a problem? That's interesting. I had interpreted it more like force-success

It's a little more complicated than that since I see there are two places where core.setFailed is being called. Do we need to call both of them? It looks like the one in runWrapper is higher up the call chain. Maybe we don't need the one in run.

I'm not sure either. I can play around with it.

Or do you mean that if we expect-error that there must be an error, otherwise there was a problem? That's interesting. I had interpreted it more like force-success

That's what I mean. If we expect an error and there is none, then that's an error. :)

I've added this logic at the end of the outer try block (indicating that an error was not thrown). Thanks!!

Interestingly, I found that the outer try block wasn't very useful. This catch block with origError is where I was able to throw/catch/stop errors appropriately. I removed the duplicate logic from the outer try block because it wasn't doing anything.

Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>

…or-input

analyze/action.yml

src/analyze-action.ts

Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>

…or-input

aeisenberg

I think all the bits are correct floating around somewhere, but there must have been a bad merge and there are some syntax errors.

src/analyze-action.ts

aeisenberg · 2022-08-16T19:40:03Z

src/analyze-action.ts

  } catch (origError) {
    const error =
      origError instanceof Error ? origError : new Error(String(origError));
    if (
-      actionsUtil.getOptionalInput("expect-error") === "false" ||
+      actionsUtil.getOptionalInput("expect-error") !== "true" ||


Maybe a bad merge here?

I think the failures are expected — I removed the ram: 1 line in the last commit to make sure that if expect-error was true and no error was thrown, that it would throw Error: expect-error input was set to true but no error was thrown.

I'll put the line back now that I have that run 👍

angelapwen · 2022-08-16T20:08:44Z

So far I have tested to confirm the following cases:

when expect-error is set on a repo that's not this one, expect-error input parameter is for internal use only. It should only be set by codeql-action or a fork. is thrown. (run)
when expect-error is set on this repo, and no error is found, expect-error input was set to true but no error was thrown. is thrown. (run)
when expect-error is set on this repo and an error is thrown, the job succeeds and is green (run)

I will test to make sure it does the right things in a fork and update.

angelapwen · 2022-08-16T20:36:44Z

The fork logic works 👍 (run)

I will do some comment and commit cleanup.

aeisenberg

This is good. It's a nice and simple solution you came up with in the end.

angelapwen · 2022-08-16T22:39:38Z

Thank you! I forgot to remove a couple of core.info() debug lines. Will remove them now, squash and merge, and then add these PR checks to the required set.

angelapwen added 2 commits August 12, 2022 11:11

Add expect-error to expected failure job check

b7d4497

Remove "failure expected" in PR check name

4fb1cdc

angelapwen marked this pull request as ready for review August 12, 2022 10:38

angelapwen requested a review from a team as a code owner August 12, 2022 10:38

angelapwen commented Aug 12, 2022

View reviewed changes

aeisenberg reviewed Aug 12, 2022

View reviewed changes

angelapwen and others added 9 commits August 12, 2022 16:41

Update expect-errors input description

c756bae

Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>

Add double quotes to input description

59d39e3

Throw error if caller isn't codeql-action

f5a4ea9

Merge remote-tracking branch 'origin/main' into angelapwen/expect-err…

14763b6

…or-input

Fix string equality comparison

e31c33d

Move error throwing location

46f3540

Add print debug messages

3606edb

Fix expected repo URL

334b121

Set failure if input is set incorrectly

65fcacd

aeisenberg reviewed Aug 16, 2022

View reviewed changes

analyze/action.yml Outdated Show resolved Hide resolved

src/analyze-action.ts Outdated Show resolved Hide resolved

src/analyze-action.ts Outdated Show resolved Hide resolved

src/analyze-action.ts Outdated Show resolved Hide resolved

angelapwen and others added 8 commits August 16, 2022 11:03

Update expect-errors input description

3d8828a

Co-authored-by: Andrew Eisenberg <aeisenberg@github.com>

Address comments, throw failure if expected and didn't fail

6be3892

Merge remote-tracking branch 'origin/main' into angelapwen/expect-err…

0736537

…or-input

Set failure in other spot for expect-error but not thrown

1e1da28

Test removing runWrapper's failure

66bfa45

Check error input is not default value

70d3299

Test remove runWrapper setting failure

41f86d9

Test run where error expected but not received

5aec66f

aeisenberg reviewed Aug 16, 2022

View reviewed changes

Fail with ram:1 in failure workflow

98d4d17

Comment fixups

f8f36a7

angelapwen requested a review from aeisenberg August 16, 2022 22:26

aeisenberg approved these changes Aug 16, 2022

View reviewed changes

Remove extraneous log lines

b67600f

angelapwen merged commit 9b7fa3d into main Aug 16, 2022

angelapwen deleted the angelapwen/expect-error-input branch August 16, 2022 23:27

This was referenced Aug 17, 2022

Merge main into releases/v2 #1189

Closed

Merge main into releases/v2 #1192

Merged

Merge releases/v2 into releases/v1 #1195

Merged

aliscco mentioned this pull request Jun 22, 2023

[Snyk] Fix for 1 vulnerabilities aliscco/codeql-action#613

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `expect-error` input to force PR check green on expected failure #1177

Add `expect-error` input to force PR check green on expected failure #1177

angelapwen commented Aug 12, 2022 •

edited

angelapwen Aug 12, 2022

aeisenberg Aug 12, 2022

angelapwen Aug 12, 2022

angelapwen Aug 12, 2022

adityasharad Aug 12, 2022

angelapwen Aug 16, 2022

aeisenberg Aug 12, 2022

aeisenberg Aug 12, 2022

angelapwen Aug 12, 2022

aeisenberg Aug 12, 2022

angelapwen Aug 16, 2022

angelapwen Aug 16, 2022

aeisenberg left a comment

aeisenberg Aug 16, 2022

angelapwen Aug 16, 2022

angelapwen commented Aug 16, 2022 •

edited

angelapwen commented Aug 16, 2022

aeisenberg left a comment

angelapwen commented Aug 16, 2022

	description: For PR checks. If true, the Action will not set Actions status to failure.
	description: [Internal] If true, the Action will not set Actions status to failure.

		@@ -183,7 +183,10 @@ async function run() {
		} catch (origError) {

Add expect-error input to force PR check green on expected failure #1177

Add expect-error input to force PR check green on expected failure #1177

Conversation

angelapwen commented Aug 12, 2022 • edited

Merge / deployment checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aeisenberg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angelapwen commented Aug 16, 2022 • edited

angelapwen commented Aug 16, 2022

aeisenberg left a comment

Choose a reason for hiding this comment

angelapwen commented Aug 16, 2022

Add `expect-error` input to force PR check green on expected failure #1177

Add `expect-error` input to force PR check green on expected failure #1177

angelapwen commented Aug 12, 2022 •

edited

angelapwen commented Aug 16, 2022 •

edited