Dependabot opens PR for unreleased github-actions

[ParanoidUser]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

github-actions

Package manager version

No response

Language version

No response

Manifest location and content before the Dependabot update

No response

dependabot.yml content

https://github.com/ParanoidUser/toolbox-opdf/blob/develop/.github/dependabot.yml
https://github.com/ParanoidUser/codewars-handbook/blob/main/.github/dependabot.yml

Updated dependency

Bumps actions/setup-java from 19eeec562b37d29a1ad055b7de9c280bd0906d8d to c3ac5dd0ed8db40fedb61c32fbe677e6b355e94c.

What you expected to see, versus what you actually saw

Everything seems to work fine yesterday, but today Dependabot opened a couple of PRs across repositories to bump the current hash to the latest committed (but unreleased) version.

Here is the commit it's referring to actions/setup-java@c3ac5dd as well as list of releases - https://github.com/actions/setup-java/releases

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

ParanoidUser/toolbox-opdf#38
ParanoidUser/codewars-handbook#177

Smallest manifest that reproduces the issue

No response

[deivid-rodriguez]

It seems that the previous commit hash was not part of any release either?

In that case, I think this is expected. Dependabot understands that you want to track unreleased versions of the action if you are already pointing to unreleased versions of the action, otherwise you would specify released versions instead.

Can you explain your expectation a bit more? Were you temporarily pointing to that commit for some reason and were expecting Dependabot to switch you back to a released version as soon as that specific commit would become part of one release?

[jas88]

This is a manifestation of actions/setup-java#422 - v 3.7.0 was released, Dependabot detected this and upgraded hundreds of repositories to use it, then the maintainers deleted the release leaving 3.6.0 as latest again.

Currently Dependabot just looks at this situation and goes "well, 3.7.0 is newer than 3.6.0, so everything is OK" instead of reacting to the fact 3.7.0 is now missing and the build is broken. Now we know this scenario is possible, could Dependabot try to handle it more gracefully please - either a PR to revert back to the now-latest release so builds work again, or at least flag the issue somewhere instead of just reporting "all ok"?

[deivid-rodriguez]

I think the author of java/setup-java might've fixed something on their side because the previous commit hash which had no associated tag a few hours ago now has 3.8.0 linked to it?

We've had requests to create downgrade PRs due to yanked releases in other ecosystems in the past too, for example #2155 or #2153. I guess that's a valid request for actions too :)

For @ParanoidUser's case, I think specifying a commit hash with a commented out version like in this feature might have avoided the issue. I will double check that, namely, that a commit hash with an incorrect version tag commented out (incorrect because of the removal of the associated tag) won't get any updates.

[jas88]

There was a call on that repo for them to fix it by retagging 3.6.0 as 3.8.0 rather than just disappearing 3.7.0 down the memory hole, presumably that's now been done, so Dependabot should now begin fixing the damaged repos automatically. Switching to using the hash would prevent this, but also prevent upgrades, so not really a solution generally AIUI?

[deivid-rodriguez]

Switching to using the hash would prevent this, but also prevent upgrades, so not really a solution generally AIUI?

How so? Dependabot still creates PRs even if you pin yourself to a commit hash.

[Gakk]

Approx. 15 minutes ago the setup-java project released version 3.8.0 😃

[deivid-rodriguez]

For @ParanoidUser's case, I think specifying a commit hash with a commented out version like in #5951 might have avoided the issue. I will double check that, namely, that a commit hash with an incorrect version tag commented out (incorrect because of the removal of the associated tag) won't get any updates.

Just looping back here to confirm that even if a commented out version is specified, one will get updates to an unreleased version, and on top of that, the version comment won't be updated.

Hard for us to fix, but we should probably do something about it at some point, at least detect the situation and avoid creating a PR.

[vy]

@deivid-rodriguez, I think, for one, the docs are misleading on this:

... this approach [using SHAs] means you will not automatically receive updates for an action ...

This is certainly not the case. We do and we should.

At Log4j, we also have the very same problem described in this ticket. As this dependabot PR shows,

apache/logging-parent/.github/workflows/build-reusable.yaml@e45457c683302242be5e8e7c3c33edf8f0e0ec0e # rel/10.4.0

(Commit e45457c683302242be5e8e7c3c33edf8f0e0ec0e indeed points to the 10.4.0 release tagged with rel/10.4.0)

reference is replaced with

apache/logging-parent/.github/workflows/build-reusable.yaml@7750fb0f674da0abb358414ea19464ecd996e36f # rel/10.4.0

Noite that neither 7750fb0f674da0abb358414ea19464ecd996e36f points to any release, nor the # rel/10.4.0 comment is correct anymore.

Hence, dependabot bumps the GitHub Actions reusable workflow reference to a commit ID that is not associated with any release.

[jakevdp]

I saw similar behavior in google/jax#20639; dependabot opened a PR changing the commit hash to the current HEAD of actions/upload-artifact, while claiming the commit was associated with v4.

[deivid-rodriguez]

... this approach [using SHAs] means you will not automatically receive updates for an action ...

This is certainly not the case. We do and we should.

@vy Sorry for never replying here. You're right, I think the docs are misleading there. What I think they want to imply is that if you specify a major version for example (not a SHA), then your actions will automatically use the latest patch/security version in that major. That won't happen if you specify a SHA: Github Actions will always pick up that specific revision of the action. I think that's what's meant by "will not automatically receive updates for an action", but it's definitely confusing since Dependabot WILL indeed create PRs with updates for the workflows using a SHA-pinned action.

I think the Dependabot team would definitely appreciate a PR rewording those docs to make things more clear!

[jakevdp]

To be clear, the issue I saw is not a documentation issue, but bug in the dependabot implementation. Dependabot claimed to be updating a commit hash to v4, when in fact it was updating to the latest (unreleased) commit on HEAD.

I posted here becuase I thought this was a duplicate issue. If you'd prefer me to open a new issue, please let me know!

[deivid-rodriguez]

@jakevdp I'm not sure. Original culprit here was action's author tagging a new version, and then deleting the tag after Dependabot has created PRs. There's not a lot that can be done about that in Dependabot's side, but maybe there are other situations where Dependabot is creating this kind of problem that can (and should) be properly addressed 🤷.

[vy]

@deivid-rodriguez, dependabot submits PRs updating SHAs to values that don't correspond to a release. As @jakevdp and I have been trying to explain, this is not a documentation issue, this is certainly unexpected, and hence pretty much qualifies as a dependabot bug.

This bug also has security implications too. Such that, OSSF Scorecards expects a project to pin dependencies using their hashes, though as we have demonstrated above, dependabot messes up all such pinning.

[deivid-rodriguez]

@vy I did not say this is a documentation issue. You pointed out that, in addition to the report, the documentation was misleading and I said you're right and recommended to update it.

As I've been trying to explain, the original reason as reported by @ParanoidUser explaining why the Dependabot PR did not correspond to a release, it's because the action's author deleted the release (tag) after the PR was created, so the SHA became "orphan". I think there's not a lot that can be done about that.

I think the other example given by @jakevdp (google/jax#20639) may be due to Dependabot not recognizing the version comment format (# ratchet: actions/upload-artifact@v4) and incorrectly thinking that the action is just tracking the latest HEAD sha (and thus updating it).

I think the latter issue can be addressed in Dependabot's side.

In any case, I'm no longer a maintainer of this repo so I can't help here, just tracking this issue and offering advice because I had participated in this thread originally when I was maintaining this repo.

[jakevdp]

may be due to Dependabot not recognizing the version comment format (# ratchet: actions/upload-artifact@v4) and incorrectly thinking that the action is just tracking the latest HEAD sha (and thus updating it).

This is not my experience in general: I use the commit hash approach everywhere in the projects I manage, and in general dependabot updates the commit hash to the one associated with the most recent release. There are many examples; one is google/jax#20520: you can see that dependabot reports updating from 5.0.0 to 5.1.0, and includes the correct commit hash associated with 5.1.0.

The issue I reported above is unique in that dependabot chose an unreleased commit, while incorrectly reporting that it was associated with the v4 tag. I'm not sure whether there was a tag created and removed in the upstream repository; it doesn't seem like there's any way to check for that. Perhaps dependabot has internal logs somewhere that could shed some light?

[deivid-rodriguez]

Right @jakevdp, then it was probably a bad guess!

Ok, I have another guess. In https://github.com/google/jax/pull/20639/files, I think the previous SHA in your workflows (5d5d22a31266ced268874388b861e4b58bb5c2f3I) became out of date (now v4 is associated to a newer SHA). That makes Dependabot is incorrectly think that you want to pin to the latest SHA in the main branch of the action and updating to that.

If that's the case, that's a Dependabot bug, it should trust the comment even if the SHA does not point to the commented tag.