[ci.jio/infra.ci.jio] datadog plugin destroyed the build data. Can we stop using it?

[dduportal]

Service(s)

ci.jenkins.io, infra.ci.jenkins.io, Datadog

Summary

The version 7.0.0 of the Jenkins datadog plugin has been released 3 days ago and we deployed it to the 2 controllers using it: ci.jenkins.io and infra.ci.jenkins.io.

Again, we see the build history destroyed:

• Builds are marked as "1 Jan 1970"
• Some builds are stuck (despite being finished since hours or days)
• We cannot stop the stuck build unless using the Console: clicking on the "stop build" only ends in an error stack in the logs:
• Any restart/reload spams the logs with XMl parsong errors like

could not load /var/jenkins_home/jobs/Tools/jobs/bom/branches/master/builds/2463
javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1]
Message: Premature end of file.
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.next(Unknown Source)
	at com.thoughtworks.xstream.io.xml.StaxReader.pullNextEvent(StaxReader.java:58)
Caused: com.thoughtworks.xstream.io.StreamException: 
	at com.thoughtworks.xstream.io.xml.StaxReader.pullNextEvent(StaxReader.java:74)
	at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)
	at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:141)
	at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:118)
	at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103)
	at com.thoughtworks.xstream.io.xml.StaxReader.<init>(StaxReader.java:45)
	at com.thoughtworks.xstream.io.xml.StaxDriver.createStaxReader(StaxDriver.java:173)
	at com.thoughtworks.xstream.io.xml.StaxDriver.createReader(StaxDriver.java:100)
	at hudson.XmlFile.unmarshal(XmlFile.java:196)
Caused: java.io.IOException: Unable to read /var/jenkins_home/jobs/Tools/jobs/bom/branches/master/builds/2463/build.xml

@MarkEWaite did warn us in https://matrix.to/#/!JLUOInpEYmxJIYXlzs:matrix.org/$X94sIpk_CEMCxnnu-JmIzR1aHg_6uG0-Crl6SMXCD_Y?via=g4v.dev&via=gitter.im&via=matrix.org about other user with the same problem. He mentionned the following loooong issue: https://issues.jenkins.io/browse/JENKINS-66328

Related: #4079

Notes

• In order to stop a "zombie" (stuck) build, an administrator need to run the following code in the Administration Console:

Jenkins.instance
.getItemByFullName('<job name>')
.getBranch('<branch or PR name')
.getBuildByNumber(<build ID>)
.finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));

The job names can be determined with:

Jenkins.instance.getAllItems(AbstractItem.class).each {
    println(it.fullName)
  };

[dduportal]

Update:

• Took 2 snapshots of the 2 controllers so we can try to restore data if fixed, or we can provide data to datadog plugin developers
• Worked on BOM testing build is stuck #4079 and stopped stuck builds
• Removed all visible stuck builds on infra.ci and triggered a new build: terrafomr jobs, kubernetes, website jobs and packer-image. There might be others

Todo:

• Clean up the ci.jenkins.io build queue by fixing each job one after the other

[dduportal]

Opened jenkinsci/datadog-plugin#423 to make the plugin maintainers aware

[dduportal]

A quick discussion/research by the team (thanks @lemeurherve !) shows the use cases we have on the Jenkins infra with the datadog plugin.
The goal for us is to understand the pros and cons of removing the plugin (but keeping the datadog-agent infra. collection: it's only the Jenkins integration through the plugin we consider disabling) to protect ourselves from such failure as it it the 3rd time in the past 2 months.

• We have a monitor of the build queue which will be impacted (as the metric won't be reported anymore): https://github.com/jenkins-infra/datadog/blob/main/datadog-monitors.tf#L194-L213
  • Removing the datadog plugin from ci.jio or infra.ci.jio would need us to disable/delete this monitor
• ci.jenkins.io utilizes the datadog CI/CD observability (with the collected metrics + traces from datadog) as per Add observability for the build agents #2769
  • Removing the datadog plugin from ci.jio would decrease the observability around the BOM. Please note that agents have their metrics and logs collected even if we remove the plugin so we won't be completely blind here
• infra.ci.jenkins.io is most probably using the plugin for the same reason as ci.jio (observability of pipelines). But don't use it: we should start considering to stop using the plugin in this private controller.

[dduportal]

Update: stopped all the apparent builds stuck on ci.jenkins.io (e.g. from the build queue, checking each build and if older thant 2 hours then check the associated branch + history)

[jonesbusy]

What about https://plugins.jenkins.io/opentelemetry/ ?

Not sure if the collector can send metrics to datadog but might be an option to keep observability of bom builds

[lemeurherve]

For the record,

The 7.0.0 release of the plugin has been removed from the Update Center

Originally posted by @nikita-tkachenko-datadog in jenkinsci/datadog-plugin#423 (comment)

[MarkEWaite]

Confirmed that it has been removed from the update center by opening

https://updates.jenkins.io/latest/datadog.hpi?mirrorlist

That URL opens to the 6.0.3 release of the plugin.

[dduportal]

For the record,

The 7.0.0 release of the plugin has been removed from the Update Center

Originally posted by @nikita-tkachenko-datadog in jenkinsci/datadog-plugin#423 (comment)

Confirmed that it has been removed from the update center by opening

https://updates.jenkins.io/latest/datadog.hpi?mirrorlist

That URL opens to the 6.0.3 release of the plugin.

As such, we'll try to revert the plugin to 6.0.3 to check if it works again without blocking builds or destroying histories.

The following operations will be performed:

• Set ci.jenkins.io in "maintenance" mode to limit the amount of changed files (which will be lost if backup is restored)
• Take a snapshot of the JENKINS_HOME's data disk to serve as backup if things go wrong
• Upload the 6.0.3 datadog plugin HPI manually from the UI and restart
• Then:
  • If The downgrade goes well and solves the 1970 (and stuck builds) builds then we continue with the plugin (for now) as we cannot spend the effort to get away from datadog on ci.jenkins.io for now, and we'll be carefull on upcoming upgrades of datadog
  • Else if if fails terribly, then we restore the backup and try the same with removing the plugin

[dduportal]

For the record,

The 7.0.0 release of the plugin has been removed from the Update Center

Originally posted by @nikita-tkachenko-datadog in jenkinsci/datadog-plugin#423 (comment)

Confirmed that it has been removed from the update center by opening
https://updates.jenkins.io/latest/datadog.hpi?mirrorlist
That URL opens to the 6.0.3 release of the plugin.

As such, we'll try to revert the plugin to 6.0.3 to check if it works again without blocking builds or destroying histories.

The following operations will be performed:

• Set ci.jenkins.io in "maintenance" mode to limit the amount of changed files (which will be lost if backup is restored)

• Take a snapshot of the JENKINS_HOME's data disk to serve as backup if things go wrong

• Upload the 6.0.3 datadog plugin HPI manually from the UI and restart

• Then:

  • If The downgrade goes well and solves the 1970 (and stuck builds) builds then we continue with the plugin (for now) as we cannot spend the effort to get away from datadog on ci.jenkins.io for now, and we'll be carefull on upcoming upgrades of datadog
  • Else if if fails terribly, then we restore the backup and try the same with removing the plugin

Update:

• Downgrade of datadog plugin done with success
• The datetime of 1970 is not solved by the downgrade.
• However no more build stuck: triggereed happenend as expected.

[dduportal]

Next step: removing datadog plugin from infra.ci.jenkins.io.

Ping @smerle33: don't forget there is a datadog configuration to remove before upgrading infra.ci with an image without the plugin (otherwise it will crash at startup).

[dduportal]

Update:

• datadog plugin has been removed from infra.ci.jenkins.io and weekly.ci.jenkins.io
  • Configuration removed in feat(infra.ci): remove datadog plugin configuration kubernetes-management#5209
  • Plugin removal: feat(weekly.ci/infra.ci): remove datadog plugin docker-jenkins-weekly#1606 (deployed with Bump Jenkins Weekly docker image version on infra.ci.jenkins.io to 1.40.4-2.456 kubernetes-management#5210)
• A fixing patch has been published on the datadog plugin: version 7.0.1. Worth upgrading on ci.jenkins.io (need planning, announce, backup)

[dduportal]

Update:

A fixing patch has been published on the datadog plugin: version 7.0.1. Worth upgrading on ci.jenkins.io (need planning, announce, backup)

This has been done with success yesterday:

• Jenkins home snapshot taken to back-up data
• Plugin upgrade went well: no more stuck builds neiter "1970" builds.

We can close this issue!