Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Have the Sidekiq process die if startup fails #3717

Closed
agrobbin opened this issue Jan 11, 2018 · 1 comment
Closed

Have the Sidekiq process die if startup fails #3717

agrobbin opened this issue Jan 11, 2018 · 1 comment

Comments

@agrobbin
Copy link

agrobbin commented Jan 11, 2018 •
edited
Loading

Ruby version: 2.5.0
Sidekiq 5.0.5 / Pro 3.5.4 / Ent 1.6.1

If Redis happens to be unavailable at startup time of the sidekiq process, currently, something like this will happen:

2018-01-08T01:20:21.218Z 1 TID-gq1xn1frs DEBUG: ReliablePush activated
2018-01-08T01:20:21.222Z 1 TID-gq1xn1frs INFO: Booting Sidekiq 5.0.5 with redis options {:namespace=>nil, :id=>"Sidekiq-server-PID-1", :url=>"redis://[redacted]:6379"}
2018-01-08T01:20:33.053Z 1 TID-gq1xn1frs WARN: {"context":"Exception during Sidekiq lifecycle event.","event":"startup"}
2018-01-08T01:20:33.053Z 1 TID-gq1xn1frs WARN: SocketError: getaddrinfo: Name or service not known
2018-01-08T01:20:33.053Z 1 TID-gq1xn1frs WARN: /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:229:in `getaddrinfo'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:229:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:313:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:336:in `establish_connection'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:101:in `block in connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:293:in `with_reconnect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:100:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/redis.rb:105:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:364:in `ensure_connected'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:221:in `block in process'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:306:in `logging'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:220:in `process'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:120:in `call'
/app/vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/redis.rb:69:in `call'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:1398:in `block in smembers'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'
/app/vendor/ruby-2.3.3/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:1397:in `smembers'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:128:in `block in listen_for_pauses'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:95:in `block in redis'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:92:in `redis'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:127:in `listen_for_pauses'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:337:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:54:in `block in fire_event'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:52:in `each'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:52:in `fire_event'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/cli.rb:92:in `run'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/bin/sidekiq:12:in `<top (required)>'
/app/vendor/bundle/ruby/2.3.0/bin/sidekiq:22:in `load'
/app/vendor/bundle/ruby/2.3.0/bin/sidekiq:22:in `<top (required)>'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
/app/vendor/bundle/ruby/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
/app/bin/bundle:3:in `load'
/app/bin/bundle:3:in `<main>'
2018-01-08T01:20:43.060Z 1 TID-gq1yrgak0 WARN: {"context":"senate"}
2018-01-08T01:20:43.060Z 1 TID-gq1yrgak0 WARN: SocketError: getaddrinfo: Name or service not known
2018-01-08T01:20:43.061Z 1 TID-gq1yrgak0 WARN: /app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:229:in `getaddrinfo'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:229:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:313:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:336:in `establish_connection'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:101:in `block in connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:293:in `with_reconnect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:100:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/redis.rb:105:in `connect'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:364:in `ensure_connected'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:221:in `block in process'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:306:in `logging'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:220:in `process'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:120:in `call'
/app/vendor/bundle/ruby/2.3.0/gems/newrelic_rpm-4.5.0.337/lib/new_relic/agent/instrumentation/redis.rb:69:in `call'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:752:in `block in set'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'
/app/vendor/ruby-2.3.3/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'
/app/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:750:in `set'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-ent-1.6.1/lib/sidekiq-ent/senate.rb:88:in `block in stage_coup!'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:95:in `block in redis'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:64:in `block (2 levels) in with'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `handle_interrupt'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:63:in `block in with'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `handle_interrupt'
/app/vendor/bundle/ruby/2.3.0/gems/connection_pool-2.2.1/lib/connection_pool.rb:60:in `with'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq.rb:92:in `redis'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-ent-1.6.1/lib/sidekiq-ent/senate.rb:87:in `stage_coup!'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-ent-1.6.1/lib/sidekiq-ent/senate.rb:99:in `cycle'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
/app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.073Z 1 TID-gq1xn1frs WARN: getaddrinfo: Name or service not known
2018-01-08T01:20:53.115Z 1 TID-gq1yrlfdc ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.116Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.116Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.142Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.142Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.142Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.142Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.143Z 1 TID-gq1yrlfdc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.145Z 1 TID-oxi6lzjko ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.147Z 1 TID-oxi6lzik0 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.148Z 1 TID-oxi6lzg0c ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.149Z 1 TID-oxi6lzg0c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.150Z 1 TID-oxi6lzidw ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.150Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.150Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.151Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.151Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.151Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.151Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.151Z 1 TID-oxi6lzidw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.153Z 1 TID-oxi6lzhvk ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.155Z 1 TID-oxi6lziw8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.157Z 1 TID-oxi6lzgio ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.158Z 1 TID-oxi6lzj2c ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.158Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.159Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.159Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.159Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.159Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.159Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.159Z 1 TID-oxi6lzj2c ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.161Z 1 TID-oxi6lzfu8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.163Z 1 TID-oxi6lzguw ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.165Z 1 TID-oxi6lzhjc ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.165Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.167Z 1 TID-oxi6lzi1o ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.168Z 1 TID-oxi6lzjek ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.170Z 1 TID-oxi6lziq4 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.171Z 1 TID-oxi6lzh74 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.173Z 1 TID-oxi6lzh10 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.174Z 1 TID-oxi6lzgos ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.176Z 1 TID-oxi6lzg6g ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.177Z 1 TID-oxi6lzj8g ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.179Z 1 TID-oxi6lzgck ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.181Z 1 TID-oxi6lzhpg ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.188Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.183Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.185Z 1 TID-oxi6lzi7s ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.187Z 1 TID-oxi6lzfo4 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.187Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.187Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.187Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.187Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.187Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.187Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.183Z 1 TID-oxi6lzhd8 ERROR: Error fetching job: undefined method `retrieve_work' for nil:NilClass
2018-01-08T01:20:53.188Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.189Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.187Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.190Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.190Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.189Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.188Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-pro-3.5.4/lib/sidekiq/pro/super_fetch.rb:32:in `retrieve_work'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.191Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.191Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzg6g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:91:in `get_one'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.192Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.193Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.193Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:101:in `fetch'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzhpg ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.194Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.194Z 1 TID-oxi6lzh74 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:84:in `process_one'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzhjc ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzi1o ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.195Z 1 TID-oxi6lzjek ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.195Z 1 TID-oxi6lziq4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzgos ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/processor.rb:73:in `run'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzi7s ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzfo4 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.196Z 1 TID-oxi6lzh10 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.197Z 1 TID-oxi6lzj8g ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.197Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:16:in `watchdog'
2018-01-08T01:20:53.197Z 1 TID-oxi6lzhd8 ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-01-08T01:20:53.197Z 1 TID-oxi6lzgck ERROR: /app/vendor/bundle/ruby/2.3.0/gems/sidekiq-5.0.5/lib/sidekiq/util.rb:25:in `block in safe_thread'

Each supporting thread (Senate election, period job scheduling, job processing, etc.) raises an exception, but the master sidekiq process continues to survive, not being able to process any work sitting in the queue(s).

We've run into issues with this in production, with our Kubernetes cluster not knowing that the sidekiq process has effectively died and needs to be restarted.
@agrobbin
Copy link
Author

Thanks for the quick fix @mperham!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@agrobbin