Race condition in shutdown

[mperham]

If a job continues processing until the shutdown timeout (-t), there is a race condition between requeueing the job and the job terminating. Specifically if it raises an error, it can create a retry after the job has been requeued, leading to job duplication.

See #4112 for a real world case.

[mperham]

Notes:

• Sidekiq does document that jobs can be run twice, this is one such case.
• There's no easy way to remove the race condition completely. Two threads can be performing two Redis operations at the same time and I don't want to dramatically increase the complexity of our Redis communications, which itself can introduce new bugs or race conditions. Lua would be necessary to make these high-level operations (e.g. bulk requeue, create retry) mutually exclusive.

[mperham]

I think this might be fixed by #4141 and 5.2.6.