Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem with redis-rb 4.2.* gem on Heroku #974

Closed
iduuck opened this issue Jan 19, 2021 · 3 comments
Closed

Problem with redis-rb 4.2.* gem on Heroku #974

iduuck opened this issue Jan 19, 2021 · 3 comments

Comments

@iduuck
Copy link

iduuck commented Jan 19, 2021

Hello, I had updated my redis gem and tried to deploy to production (on staging everything went fine – however, staging is not using rediss protocol).

Before with redis-rb 4.1.* everything was working fine, so I thought this could be related to the gem rather than the server.

Backtrace

2021-01-19T11:16:21.714945+00:00 app[worker.1]: pid=4 tid=xx INFO: Booting Sidekiq 6.1.2 with redis options {:url=>"rediss://:REDACTED@xxx-xx-xxx-xxx-xx.eu-west-x.compute.amazonaws.com:xxxx"}
2021-01-19T11:16:21.735494+00:00 app[worker.1]: pid=4 tid=xx WARN: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
2021-01-19T11:16:21.735577+00:00 app[worker.1]: pid=4 tid=xx WARN: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect_nonblock'
2021-01-19T11:16:21.735578+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect'
2021-01-19T11:16:21.735579+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/connection/ruby.rb:302:in `connect'
2021-01-19T11:16:21.735579+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:354:in `establish_connection'
2021-01-19T11:16:21.735580+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:112:in `block in connect'
2021-01-19T11:16:21.735580+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
2021-01-19T11:16:21.735581+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:111:in `connect'
2021-01-19T11:16:21.735581+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:386:in `ensure_connected'
2021-01-19T11:16:21.735581+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
2021-01-19T11:16:21.735582+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
2021-01-19T11:16:21.735582+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
2021-01-19T11:16:21.735584+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis/client.rb:131:in `call'
2021-01-19T11:16:21.735584+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:2192:in `block in hget'
2021-01-19T11:16:21.735585+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
2021-01-19T11:16:21.735585+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
2021-01-19T11:16:21.735586+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:69:in `synchronize'
2021-01-19T11:16:21.735586+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/redis-4.2.5/lib/redis.rb:2191:in `hget'
2021-01-19T11:16:21.735587+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:361:in `block in status_from_redis'
2021-01-19T11:16:21.735587+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.1.2/lib/sidekiq.rb:98:in `block in redis'
2021-01-19T11:16:21.735588+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:63:in `block (2 levels) in with'
2021-01-19T11:16:21.735588+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `handle_interrupt'
2021-01-19T11:16:21.735589+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:62:in `block in with'
2021-01-19T11:16:21.735589+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `handle_interrupt'
2021-01-19T11:16:21.735589+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/connection_pool-2.2.3/lib/connection_pool.rb:59:in `with'
2021-01-19T11:16:21.735590+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.1.2/lib/sidekiq.rb:95:in `redis'
2021-01-19T11:16:21.735590+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:360:in `status_from_redis'
2021-01-19T11:16:21.735590+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:272:in `initialize'
2021-01-19T11:16:21.735591+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:187:in `new'
2021-01-19T11:16:21.735591+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:187:in `block in load_from_array'
2021-01-19T11:16:21.735592+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:186:in `each'
2021-01-19T11:16:21.735592+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:186:in `load_from_array'
2021-01-19T11:16:21.735593+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:157:in `load_from_hash'
2021-01-19T11:16:21.735594+00:00 app[worker.1]: /app/config/initializers/sidekiq.rb:6:in `<main>'
2021-01-19T11:16:21.735595+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
2021-01-19T11:16:21.735595+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/bootsnap-1.5.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'
2021-01-19T11:16:21.735596+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `block in load'
2021-01-19T11:16:21.735596+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-01-19T11:16:21.735596+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:318:in `load'
2021-01-19T11:16:21.735597+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/engine.rb:666:in `block in load_config_initializer'
2021-01-19T11:16:21.735603+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/appsignal-2.11.3/lib/appsignal/hooks/active_support_notifications.rb:21:in `block in instrument'
2021-01-19T11:16:21.735603+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2021-01-19T11:16:21.735604+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/appsignal-2.11.3/lib/appsignal/hooks/active_support_notifications.rb:20:in `instrument'
2021-01-19T11:16:21.735604+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/engine.rb:665:in `load_config_initializer'
2021-01-19T11:16:21.735605+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
2021-01-19T11:16:21.735605+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/engine.rb:624:in `each'
2021-01-19T11:16:21.735605+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/engine.rb:624:in `block in <class:Engine>'
2021-01-19T11:16:21.735606+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:32:in `instance_exec'
2021-01-19T11:16:21.735606+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:32:in `run'
2021-01-19T11:16:21.735606+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:61:in `block in run_initializers'
2021-01-19T11:16:21.735607+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
2021-01-19T11:16:21.735607+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
2021-01-19T11:16:21.735608+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
2021-01-19T11:16:21.735608+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
2021-01-19T11:16:21.735608+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
2021-01-19T11:16:21.735609+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:50:in `each'
2021-01-19T11:16:21.735609+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:50:in `tsort_each_child'
2021-01-19T11:16:21.735610+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:415:in `call'
2021-01-19T11:16:21.735610+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
2021-01-19T11:16:21.735610+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
2021-01-19T11:16:21.735611+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:347:in `each'
2021-01-19T11:16:21.735611+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:347:in `call'
2021-01-19T11:16:21.735612+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
2021-01-19T11:16:21.735612+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
2021-01-19T11:16:21.735613+00:00 app[worker.1]: /app/vendor/ruby-2.6.6/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
2021-01-19T11:16:21.735613+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/initializable.rb:60:in `run_initializers'
2021-01-19T11:16:21.735613+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/railties-6.0.3.4/lib/rails/application.rb:363:in `initialize!'
2021-01-19T11:16:21.735614+00:00 app[worker.1]: /app/config/environment.rb:7:in `<top (required)>'
2021-01-19T11:16:21.735614+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:324:in `require'
2021-01-19T11:16:21.735615+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:324:in `block in require'
2021-01-19T11:16:21.735615+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:291:in `load_dependency'
2021-01-19T11:16:21.735615+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.3.4/lib/active_support/dependencies.rb:324:in `require'
2021-01-19T11:16:21.735616+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.1.2/lib/sidekiq/cli.rb:252:in `boot_application'
2021-01-19T11:16:21.735616+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.1.2/lib/sidekiq/cli.rb:37:in `run'
2021-01-19T11:16:21.735617+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/gems/sidekiq-6.1.2/bin/sidekiq:31:in `<top (required)>'
2021-01-19T11:16:21.735617+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `load'
2021-01-19T11:16:21.735617+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `<top (required)>'
...
@iduuck iduuck mentioned this issue Jan 19, 2021
Closed
@byroot
Copy link
Collaborator

byroot commented Jan 19, 2021

Yes, because of a bug, SSL certs weren't verified. It was fixed in #900

So you need to either fix your SSL certificates to no longer be self signed, or if that's actually expected, then you need to pass the correct ssl_params to allow that cert.

@byroot byroot closed this as completed Jan 19, 2021
@byroot
Copy link
Collaborator

byroot commented Jan 19, 2021

See OpenSSL doc: https://www.rubydoc.info/stdlib/openssl/OpenSSL/SSL/SSLContext

You can restore the old behavior with ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }, but I don't recommend it.

@iduuck
Copy link
Author

iduuck commented Jan 19, 2021

Thanks, @byroot.

Just for completion, for everyone bumping into this issue here. Heroku is actually documenting to set the verify_mode to OpenSSL::SSL::VERIFY_NONE in the settings.

See here: https://devcenter.heroku.com/articles/heroku-redis#connecting-in-rails

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@byroot @iduuck