removing the central cache has seemingly broken dependabot (sometimes!)

[jtnord]

Service(s)

Artifactory

Summary

jenkinsci/kubernetes-client-api-plugin#247 (comment)

dependabot is looking for updates in repos - it does this by using the list of repositories and obtaining the maven-metadata.xml for each in turn.

it appears as though as soon as a repository returns 200 it no longer consults other repositories.

in the above ticket it can be seen that the metadata for io.fabric8:kubernetes-client is contained https://repo.jenkins-ci.org:443/public/io/fabric8/kubernetes-client/maven-metadata.xml (it is a snapshot version).
this confuses DB as it thinks that is all that is available and no longer proposes updates from central.

Whilst this appears to be a DB bug (the metadata being served is only SNAPSHOTs) this could have a wider impact for our API like plugins so filing here incase there is somethinh we can do.

Reproduction steps

publish a snapshot to repo.jenkins-ci.org for an artifact that is public. (an old version will do)
sit back and wait for the maintainer to publish a new release to central
sit back and wait for dependabot to not propose an update.

notice it will not arrive.

[jtnord]

nb: I filed a ticket for DB (which is private and can not share :-/ https://support.github.com/ticket/personal/0/2551888)

[jtnord]

Response from GitHub.

Hello James,

Thank you for contacting GitHub support. I spoke with our engineers and they said that Dependabot was working as expected, but you could submit a feature request to change the current behavior in our Community discussions.

They said to have Dependabot check Maven central first, you could add it to your pom.xml file before the Jenkins repository or remove the stale entry from the jenkins-ci maven repo.

Please let me know if there is anything else I can help you with.
Kind regards

[MarkEWaite]

They said to have Dependabot check Maven central first, you could add it to your pom.xml file before the Jenkins repository or remove the stale entry from the jenkins-ci maven repo.

That seems complicated. Does that mean that the repo.jenkins-ci.org response (HTTP 200) will confuse dependabot if a snapshot release was published to repo.jenkins-ci.org for any artifact that hosts its releases on Maven central?

Is there a reasonable way to configure the artifact repository to never respond with a snapshot release? If that configuration exists and we enable it, would that have other side effects?

Is there a reasonable way to identify existing artifacts that are officially distributed by Maven central but have a snapshot published on repo.jenkins-ci.org?

[jtnord]

They said to have Dependabot check Maven central first, you could add it to your pom.xml file before the Jenkins repository or remove the stale entry from the jenkins-ci maven repo.

That seems complicated.

indeed - I would say that DB is broken... the amount of artifacts that this can affect is huge - alsmost anything that is published to a different repo will by its nature have non central first.

Does that mean that the repo.jenkins-ci.org response (HTTP 200) will confuse dependabot if a snapshot release was published to repo.jenkins-ci.org for any artifact that hosts its releases on Maven central?

yup - that is exactly the case here I beleive.

Is there a reasonable way to configure the artifact repository to never respond with a snapshot release? If that configuration exists and we enable it, would that have other side effects?

Yes - but then we would need a different proxy group for snapshots and to publish new parent poms.

Is there a reasonable way to identify existing artifacts that are officially distributed by Maven central but have a snapshot published on repo.jenkins-ci.org?

not just a snapshot - anything that had a private build.

[jtnord]

you could add it to your pom.xml file before the Jenkins repository or remove the stale entry from the jenkins-ci maven repo.

clearly there is a misunderstanding of Maven by the support staff as you never add central as a repository - it is implicit. and as for removing a stale entry - if it was a release version that would break things that still needed that version. (you never delete a version). but anyway...

[MarkEWaite]

Any recommendation for next steps? Some alternatives:

• Accept that sometimes we'll miss dependabot updates
• Ask to delete snapshots from repo.jenkins-ci.org when we detect the issue
• Switch to Renovate

[jtnord]

Any recommendation for next steps.

Not sure I have any. I filed this mainly as an awareness task rather than expecting it to be solved.
I'm not sure if we need to let developers know about the issue and consider this done?

[daniel-beck]

Same problem with Mina thanks to https://repo.jenkins-ci.org/public/org/apache/sshd/sshd-core/maven-metadata.xml existing.

Dependabot CLI log

updater | 2024/03/27 18:55:12 INFO Checking if org.apache.sshd:sshd-common 2.11.0 needs updating
  proxy | 2024/03/27 18:55:12 [026] GET https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:12 [026] 200 https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:13 [028] GET https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:13 [028] 404 https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:13 [030] GET https://repo.jenkins-ci.org:443/incrementals/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:13 [030] 404 https://repo.jenkins-ci.org:443/incrementals/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:14 [032] GET https://repo.maven.apache.org:443/maven2/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:14 [032] 200 https://repo.maven.apache.org:443/maven2/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:14 [034] HEAD https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-common/2.12.1/sshd-common-2.12.1.jar
  proxy | 2024/03/27 18:55:14 [034] 404 https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-common/2.12.1/sshd-common-2.12.1.jar
  proxy | 2024/03/27 18:55:15 [036] HEAD https://repo.jenkins-ci.org:443/incrementals/org/apache/sshd/sshd-common/2.12.1/sshd-common-2.12.1.jar
  proxy | 2024/03/27 18:55:15 [036] 404 https://repo.jenkins-ci.org:443/incrementals/org/apache/sshd/sshd-common/2.12.1/sshd-common-2.12.1.jar
  proxy | 2024/03/27 18:55:16 [038] HEAD https://repo.maven.apache.org:443/maven2/org/apache/sshd/sshd-common/2.12.1/sshd-common-2.12.1.jar
  proxy | 2024/03/27 18:55:16 [038] 200 https://repo.maven.apache.org:443/maven2/org/apache/sshd/sshd-common/2.12.1/sshd-common-2.12.1.jar
updater | 2024/03/27 18:55:16 INFO Latest version is 2.12.1
  proxy | 2024/03/27 18:55:16 [040] GET https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:16 [040] 200 https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:16 [042] GET https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:16 [042] 200 https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:16 [044] GET https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:16 [044] 200 https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:16 [046] GET https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:16 [046] 404 https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:17 [048] GET https://repo.jenkins-ci.org:443/incrementals/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:17 [048] 404 https://repo.jenkins-ci.org:443/incrementals/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:17 [050] GET https://repo.maven.apache.org:443/maven2/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:17 [050] 200 https://repo.maven.apache.org:443/maven2/org/apache/sshd/sshd-common/maven-metadata.xml
  proxy | 2024/03/27 18:55:17 [052] GET https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:17 [052] 200 https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:17 [054] GET https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-core/maven-metadata.xml
  proxy | 2024/03/27 18:55:18 [054] 200 https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-core/maven-metadata.xml
updater | 2024/03/27 18:55:18 INFO Requirements to unlock update_not_possible
updater | 2024/03/27 18:55:18 INFO Requirements update strategy 
updater | 2024/03/27 18:55:18 INFO No update possible for org.apache.sshd:sshd-common 2.11.0
updater | 2024/03/27 18:55:18 INFO Checking if org.apache.sshd:sshd-core 2.11.0 needs updating
  proxy | 2024/03/27 18:55:18 [056] GET https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:18 [056] 200 https://repo.jenkins-ci.org:443/public/org/jenkins-ci/jenkins/1.112/jenkins-1.112.pom
  proxy | 2024/03/27 18:55:18 [058] GET https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-core/maven-metadata.xml
  proxy | 2024/03/27 18:55:18 [058] 200 https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-core/maven-metadata.xml
updater | 2024/03/27 18:55:18 INFO Filtered out 1 pre-release versions
  proxy | 2024/03/27 18:55:18 [060] HEAD https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-core/0.11.0-sshd-314-1/sshd-core-0.11.0-sshd-314-1.jar
  proxy | 2024/03/27 18:55:18 [060] 200 https://repo.jenkins-ci.org:443/public/org/apache/sshd/sshd-core/0.11.0-sshd-314-1/sshd-core-0.11.0-sshd-314-1.jar
updater | 2024/03/27 18:55:18 INFO Latest version is 0.11.0-sshd-314-1
updater | 2024/03/27 18:55:18 INFO No update needed for org.apache.sshd:sshd-core 2.11.0

[basil]

Any recommendation for next steps? […] Ask to delete snapshots from repo.jenkins-ci.org when we detect the issue

Deleting snapshots from Artifactory when we detect the issue seems like a reasonable next step, especially if the snapshots are over a year old. That wouldn't have helped in this case, though, since there we have a non-snapshot:

https://repo.jenkins-ci.org/public/org/apache/sshd/sshd-core/0.11.0-sshd-314-1/

When publishing our own releases of upstream software, I would recommend against using the same group ID and artifact ID, but that doesn't help in the case of existing old releases. We could delete them (that one is from 2014), but this might risk breaking old builds. If the artifact is old enough, maybe we can accept that for the sake of cleaning up our repository.

[daniel-beck]

As far as I can tell this is caused by dependabot/dependabot-core#5872 potentially misinterpreting the Maven docs ("for artifacts" might not include maven-metadata.xml?), but not entirely sure.