Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ERR invalid password using Sentinels on Sidekiq 6.0.4+ #4499

Closed
plribeiro3000 opened this issue Mar 21, 2020 · 9 comments
Closed

ERR invalid password using Sentinels on Sidekiq 6.0.4+ #4499

plribeiro3000 opened this issue Mar 21, 2020 · 9 comments

Comments

@plribeiro3000
Copy link

Ruby version:
Ruby 2.6.5 / Rails 5.2.4.2 / Sidekiq 6.04+

Initializer

require 'sidekiq'
require 'sidekiq/middleware/i18n'
require_relative '../../lib/configuration'

Sidekiq.configure_client do |config|
  config.redis = Configuration.sidekiq_client
end

Sidekiq.configure_server do |config|
  config.redis = Configuration.sidekiq_server

  ar_configuration = ActiveRecord::Base.configurations[Rails.env]
  ar_configuration['pool'] = 13

  ActiveRecord::Base.establish_connection(ar_configuration)
end

lib/configuration.rb

class Configuration
  def sidekiq_server
    {
        driver: :hiredis,
        url: "redis://<user>:<password>@<name>",
        sentinels:
          [
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
          ],
        failover_reconnect_timeout: 20
      }
  end

  def sidekiq_client
    {
        driver: :hiredis,
        url: "redis://<user>:<password>@<name>",
        size: <size>
        sentinels:
          [
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
          ],
        failover_reconnect_timeout: 20
      }
  end
end

I was using sidekiq 6.0.1 with this code without any issues. Once i've updated to 6.0.5 i've got this error:

2020-03-21T00:39:52.247Z pid=39153 tid=owdxoxoo1 INFO: Booted Rails 5.2.4.2 application in development environment
2020-03-21T00:39:52.248Z pid=39153 tid=owdxoxoo1 INFO: Running in ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
2020-03-21T00:39:52.248Z pid=39153 tid=owdxoxoo1 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2020-03-21T00:39:52.249Z pid=39153 tid=owdxoxoo1 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
ERR invalid password
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:126:in `call'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:107:in `block in connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:306:in `with_reconnect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:105:in `connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:381:in `ensure_connected'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:125:in `call'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:603:in `block in resolve_master'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:585:in `block in sentinel_detect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:576:in `each'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:576:in `sentinel_detect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:602:in `resolve_master'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:565:in `resolve'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:344:in `establish_connection'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:106:in `block in connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:306:in `with_reconnect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:105:in `connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:381:in `ensure_connected'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:125:in `call'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:286:in `block in info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:52:in `block in synchronize'
/Users/plribeiro3000/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:52:in `synchronize'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:285:in `info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:118:in `block in redis_info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:97:in `block in redis'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:94:in `redis'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:112:in `redis_info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq/cli.rb:61:in `run'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/bin/sidekiq:12:in `<top (required)>'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/sidekiq:23:in `load'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/sidekiq:23:in `<main>'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/ruby_executable_hooks:24:in `eval'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/ruby_executable_hooks:24:in `<main>'

I've tried to use master but it did not solve it.
6.0.4 has the same issue.
It works on 6.0.3
@plribeiro3000
Copy link
Author

I've tried to find what changed that could be the cause without success.
v6.0.3...master
What am i missing?

@mperham
Copy link
Collaborator

mperham commented Mar 23, 2020

Seems like a Redis config issue, not clear this is a Sidekiq issue.

@mperham mperham closed this as completed Mar 23, 2020
@plribeiro3000
Copy link
Author

@mperham I'm using the latest version of hiredis (0.6.3) and latest version of redis (4.13) for a while now and didn't change anything for the code to break besides sidekiq version.

I know this seems like a config issue or either a question, but it is not.
With sidekiq 6.0.3 it just works. If i bump just sidekiq it starts to fail.

Could you at least point me at where i should look?

@mperham
Copy link
Collaborator

mperham commented Apr 1, 2020

#4354?

@plribeiro3000
Copy link
Author

This was my first check, but it only affects logging. =s

I will keep it locked to version 6.0.3 for now until i figure out what is happening.

@mperham
Copy link
Collaborator

mperham commented Apr 1, 2020 via email

mperham added a commit that referenced this issue Apr 1, 2020
@mperham
Deep clone options, fixes #4499
3f9c4bf 
Ensures any password mangling doesn't affect the runtime options
@mperham
Copy link
Collaborator

mperham commented Apr 1, 2020

@plribeiro3000 Thank you for following up! I forgot about the change and didn't realize the codebase had special handling for sentinel passwords. You can run master if you need this fix immediately.

@plribeiro3000
Copy link
Author

Awesome. Thank you for your time to fix It!
I will check it next week.

@plribeiro3000
Copy link
Author

Sorry for the huge delay here @mperham.

Your fix did work! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mperham @plribeiro3000