Tracking deployments automatically in metrics

[janko]

We're using Elastic APM, which also supports tracking deployments that are then shown in the graphs. What I like about it is that we don't need to explicitly register a deployment, we just configure the current app version, and any time we deploy with a bumped version, the first transaction that's sent to Elastic APM with the new version will be marked as a deployment.

It's not 100% accurate, because of if there are no transactions some time after deployment, then the deployment will be marked later. But that doesn't really happen on apps that receive regular traffic. I think it's really convenient, because don't need to run anything additional during deployment.

Do you think Sidekiq metrics could work similarly? I don't know how easy it would be to implement.

[mperham]

It's hard to distinguish between a deploy and a process restart without making assumptions or requiring tool-specific knowledge. How do you tell, on startup, that the app code has been deployed? That's why we provide an API but don't built in auto-deploy "smarts". A tool like Kamal would be a good integration point.

[janko]

Sidekiq could store the last known app version in Redis that it received from metrics' metadata, and when it receives a new metrics datapoint with a different version, it would consider that a deploy and mark it as such. The user would set the app version in Sidekiq config.

[mperham]

There are still edge cases which can lead to a phantom deploy: Redis could be flushed. Race conditions between processes starting on different machines, split brain deploys (where a small percentage of a cluster is running a different deploy for testing). IMO It's better to provide an API where the user can tell us explicitly when a deploy happens rather than trying to be too clever.