Overcome increasing deviation of job retry delay when retrying linearly

[lostie]

The app that I'm working on has a few jobs that retry after a constant number of minutes. In some cases, jobs decrease their retry frequency after a certain number of retries. This is done by configuring the sidekiq_retry_in block.

Taking a look at a job that retries roughly every 5 minutes, I found out that some retries happened only after 10 minutes. After some investigation, I've tracked this variability down to these lines

sidekiq/lib/sidekiq/job_retry.rb

Lines 187 to 192 in 312672d

	jitter = rand(10) * (count + 1)
	retry_at = Time.now.to_f + delay + jitter
	payload = Sidekiq.dump_json(msg)
	redis do |conn|
	conn.zadd("retry", retry_at.to_s, payload)
	end

I understand the reasoning behind spreading the retries but when I've plotted the variability into a chart, I've noticed that the jitter value after some retries surpasses the original retry delay:

From the chart, after about 30 retries the jitter value matches the retry delay. After about 60 retries, the jitter value is more than double the retry delay.
This job halves its retry frequency after 80 retries. Even in that case, the chart shows a lot of retries where the jitter value is greater than the retry delay.

Taking inspiration from the sidekiq_retry_in block, I've created a draft:

JoinRaylo/sidekiq#1

where the retry jitter is configurable to go hand in hand with the retry delay and allows to overcome the increasing deviation shown in the chart. The block can use the retry count and the delay to calculate the jitter and falls back to the original implementation if the block is not provided.

What do you think of this approach?

[mperham]

No more configuration. Sidekiq was not designed to retry jobs 60-80 times. Why are you using linear backoff?

[lostie]

The linear backoff is a business requirement. The issue here is that the jitter is not just adding some variance to the retry delay, but instead over time it has the same or higher importance than that of the retry delay (around 20 retries the chart is already showcasing that behaviour).

[mperham]

Correct, that's how it's designed to work. If it doesn't work on the 30th try, it probably won't work on the 31st either so Sidekiq won't retry as frequently. No plans to change this.