Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

until and while executing with different arguments for client/server #742

Open
capsterx opened this issue Nov 20, 2022 · 2 comments
Open

Comments

@capsterx
Copy link

Is your feature request related to a problem? Please describe.
We have a worker process that updates a block of data, we were using :until_and_while_executing and it worked as expected. Recently we made an optimization to pass the worker arguments which allows it to limit what it looks at to update. We only want one process to run at a time so a workaround was to use just use while executing and ignore args.

Describe alternatives you've considered
I read the documentation about different args in server/client context.

  1. sidekiq unique only checks arguments client-side and they are restored from a hash server side
  2. Sidekiq::ProcessSet.new.size for my setup is the same on client and server.

Describe the solution you'd like

We want a lock like until executing that works normally and a lock while executing that limits to one per class and ignores arguments.

This is what I ended up implementing
module Locks
  class UntilAndWhileExecutingServerArgs < SidekiqUniqueJobs::Lock::UntilAndWhileExecuting                 
    include SidekiqUniqueJobs                                                                              
    
    def runtime_lock
      new_item = item.dup
      new_item[LOCK_ARGS] = Object.const_get(item[CLASS])                                                  
        .sidekiq_options[LOCK_ARGS_METHOD]
        .call(item[ARGS])
      new_item[LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(new_item)                                 
      @runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(new_item, callback, redis_pool)        
    end             
  end                 
end   

And for the worker

class Worker
   include Sidekiq::Worker
   sidekiq_options on_conflict: { server: :reschedule },
                  lock: :until_and_while_executing_server_args,
                  lock_args_method: lambda { |args|
                    if Sidekiq.server?
                      []
                    else
                      args
                    end
                  }
end

And the config
SidekiqUniqueJobs.configure do |config|
config.add_lock :until_and_while_executing_server_args, Locks::UntilAndWhileExecutingServerArgs
end

Additional context
I wasn't sure if it would have been better to get server/client args both in the client side or allow it to run as I have in the sidekiq process. I also know this code is limited to a proc, I tried calling SidekiqUniqueJobs::LockArgs.call(new_item) but that didn't work for me for some reason, not sure if i was doing something wrong or not, but I went with just directly calling it (might have been an issue with me checking stuff in pry in sidekiq?). I also originally passed a ruby kw argument to the lambda until I found Sidekiq.server?. I assume that always works to check if it's running in sidekiq.

@capsterx
Copy link
Author

We had to roll this back as it was giving an error
gems/sidekiq-unique-jobs-7.1.8/lib/sidekiq_unique_jobs/lock/until_and_while_executing.rb:49 rescue in execute
ems/sidekiq-unique-jobs-7.1.8/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb:291 reflect
ArgumentError · wrong number of arguments (given 2, expected 0)

@capsterx
Copy link
Author

capsterx commented Nov 21, 2022

I had to change the code, one including SidekiqUniqueJobs got me variables, but imported a reflect that was not compatible. Also using Sidekiq.server? prevented reschedule from working because it was called from sidekiq

module Locks
  class UntilAndWhileExecutingServerArgs < SidekiqUniqueJobs::Lock::UntilAndWhileExecuting                 
    class RuntimeLock < SidekiqUniqueJobs::Lock::WhileExecuting                                            
    def runtime_lock
      new_item = item.dup
      new_item[SidekiqUniqueJobs::LOCK_ARGS] = Object.const_get(item[SidekiqUniqueJobs::CLASS])            
        .sidekiq_options[SidekiqUniqueJobs::LOCK_ARGS_METHOD]
        .call(item[SidekiqUniqueJobs::ARGS], side: :server)
      new_item[SidekiqUniqueJobs::LOCK_DIGEST] = SidekiqUniqueJobs::LockDigest.call(new_item)              
      @runtime_lock ||= SidekiqUniqueJobs::Lock::WhileExecuting.new(new_item, callback, redis_pool)                                    
    end               
  end               
end  
class Worker
   include Sidekiq::Worker
   sidekiq_options on_conflict: { server: :reschedule },
                  lock: :until_and_while_executing_server_args,
                  lock_args_method: lambda { |args, side: :client|
                    if side == :server
                      []
                    else
                      args
                    end
                  }
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant