Better handling of Redis connection issues

[rgalanakis]

Redis has connection problems, for example:

https://lithic-technology.sentry.io/issues/4264119348
https://lithic-technology.sentry.io/issues/4264118801

Redis isn't available so the job is totally lost. DurableJobs sees this, and adds the job as a DeadSet job. We need to replay these manually.

Instead, DurableJobs should move jobs that fail due to connection errors in the Redis connection layer into the RetrySet so it's automatically retried.

The reset happens in redis/connection/ruby.rb in connect_nonblock at line 264

Migrated from webhookdb-api#707. Opened by @sentry-io[bot]

Opened at: 2023-06-21T05:58:21Z
Closed at: 2023-06-29T18:42:13Z

[rgalanakis]

Sentry issue: WEBHOOKDB-API-J6

Original Comment by @sentry-io[bot]

[rgalanakis]

Possibly fixed in #718

Original Comment by @rgalanakis

[rgalanakis]

Or in #719

Original Comment by @rgalanakis

[rgalanakis]

We can try upgrading Sidekiq to fix this; this should not be a change in Durable Jobs, this is a bug with infra or libraries, and we should not hide that.

Original Comment by @rgalanakis

[rgalanakis]

Heroku support said to try updating the timeout to 25 hours, as per redis/redis-rb#1107 (comment)

heroku redis:timeout redis-curved-26048 --seconds=90000 --app=webhookdb-api-production

Original Comment by @rgalanakis

[rgalanakis]

Closing this for now but will reopen if not fixed.

Original Comment by @rgalanakis