BUG: Token-Permissions check not reducing score for job-level contents: write permission

[diogoteles08]

Describe the bug

In the scenario of a workflow with read-only top level permissions but contents: write as a job-level permissions, no score is being reduced on the Token-Permission check.

Reproduction steps

Run the following scorecard evaluation and note that it gets a 10/10 on Token-Permissions.

scorecard --repo=diogoteles08/testing-token-permissions --checks=Token-Permissions --commit=e79c59d9c2ab27853fb76389f1203ca3f9832b3a

The code analyzed have this highly vulnerable workflow call:

permissions: read-all

jobs:
  third-party-action-with-contents-write:
    runs-on: ubuntu-latest
    name: Call third party action with write permission
    permissions:
      contents: write

    steps:
      - name: "Checkout code"
        uses: not-really-safe-source/arguably-dangerous-action@master

This issue occurs in both version 4.10.2 and 4.10.5

Expected behavior

The Token-Permission check should detect the permissive contents: write permission and the score should have a large reduction.

Additional context

I'll assign myself on this issue

[evverx]

It kind of contradicts #2338.

[evverx]

FWIW I think scorecard should actually flag "arguably dangerous actions" if they have that sort of permissions but I seriously doubt that the scorecard project has the resources to audit all the actions (and keep doing that every time they or their dependencies get updated) (and that was kind of confirmed in #2338).

[evverx]

not-really-safe-source

It doesn't matter where it comes from. It can come from Google and be "arguably dangerous". Those decisions shouldn't be based on the origin. They should be based on actual reviews/audits/scans that have to be conducted every time stuff gets updated.

Edit: looks like scorecard can use something like ossf/scorecard-action#1107 to evaluate actions (whether it's adequate or not is another matter). Anyway I really hope that this check doesn't boil down to "stuff from GitHub/Google/... is implicitly trusted and everything else is "third-party".

[diogoteles08]

I created this issue because I thought it was a simple bug, but now I realize that it was intentional and it's part of a broader discussion.

@evverx I agree with all your points.

I'll meet with some scorecard maintainers and discuss some ideas I have on how to handle this.

[evverx]

FWIW I ran scorecard against the action I don't particularly like and it got "4.1" so it seems in that particular case scorecard could have issued a warning along the lines of "this stuff isn't actively maintained and should probably be avoided". That said low scores don't necessarily mean that actions are insecure so I'm not sure how to distinguish useful suggestions from false positives (but as far as I can remember something like that is being discussed in some other issues).

[diogoteles08]

I'll close this issue as it's an expected behavior, as discussed in the issue #2338 and implemented on the PR #2355

[diogoteles08]

I created the issue #3022 discussing another idea around the problem commented here.