RTT warning can signal CPU saturation

[mperham]

I wanted to create a note regarding behavior that several Sidekiq users have seen recently. The new Redis RTT warning in Sidekiq v6.2.x can trigger if you greatly overload Sidekiq with work. Because it is CPU overloaded, other threads might not be able to get CPU time quickly and that can look like Redis network latency.

The solution is:

1. Reduce your concurrency. Default is 10. If you raised it higher, use the default. If you have a queue for CPU heavy jobs, reduce the process concurrency accordingly. You might need to set it to 1 if you are doing some heavy processing.
2. Call Thread.pass occasionally in your inner loop. This tells MRI to schedule other threads, allowing time for housekeeping and other Sidekiq internal threads (e.g. cron, job scheduling).

[Sharor]

I apologize if it's super obvious, but I don't understand what the inner loop is (in reference to point 2)?
I am working with a rails app, so does it refer to the worker or sidekiq.rb in initializers?

Reducing concurrency to 1 did not solve the problem, so I would like to try option 2.

Thank you for putting this comment in the logs either way 🙂

[nertzy]

I believe that @mperham meant that if you have a CPU-heavy operation, it likely involves some sort of looping, perhaps deeply nested inside other loops.

You can call Thread.pass somewhere in the innermost loop so that your process regularly signals to the Ruby interpreter that it's OK to pause for an instant to allow other threads that are waiting to run before resuming.

[ioanniskousis]

@mperham is Thread.pass equivalent to sleep x.x as a solution ?

[mperham]

Thread.pass is better than sleep in this case. Sleep means the thread absolutely cannot be scheduled for that time. Thread.pass just means "give other threads a chance to run if necessary". The thread can be immediately rescheduled if there's no other contention.

[ioanniskousis]

@mperham thanks for the response 👍
Is this Thread.pass applicable in an each-loop called from the Sidekiq worker ?

[nertzy]

@ioanniskousis if you're asking about whether it matters where Thread.pass is called (which class / method / source file), the answer is that is does not matter.

Whether your Sidekiq worker calls Thread.pass directly or indirectly (via other method calls in the stack), the Ruby scheduler will get a chance to temporarily pause execution and pass work to other threads.

[jpriollaud]

@ioanniskousis do you mean you will call Thread.pass instead of sleep, conditionally: if a defined waiting duration is not yet fulfilled? In other words, how are you handling the X.X in your example?

is Thread.pass until condition the idea here?

[echan00]

Also running into this problem in my application. I have concurrency set to 1 and also running Thread.pass in my loop but no luck, any other suggestions?

[kapso]

Also seeing this issue on Heroku...

• This is on Performance-M dynos (2.5 GB RAM), and are not seeing really high "Dyno Load" numbers - https://devcenter.heroku.com/articles/metrics#dyno-load
• Concurrency is 3
• We are using the latest Pro version of Sidekiq

pid=3 tid=1zhpf WARN: Your Redis network connection is performing extremely poorly.
2022-06-02T20:48:46.216903+00:00 app[worker_ingest_exception.2]: Last RTT readings were [200104, 200776, 300183, 299979, 200078], ideally these should be < 1000.
2022-06-02T20:48:46.216904+00:00 app[worker_ingest_exception.2]: Ensure Redis is running in the same AZ or datacenter as Sidekiq.
2022-06-02T20:48:46.216904+00:00 app[worker_ingest_exception.2]: If these values are close to 100,000, that means your Sidekiq process may be
2022-06-02T20:48:46.216905+00:00 app[worker_ingest_exception.2]: CPU-saturated; reduce your concurrency and/or see https://github.com/mperham/sidekiq/discussions/5039

[mperham]

If your job saturates the CPU, even a concurrency of 1 can trigger the error (although it's less likely).

[kapso]

Thanks for the quick response 🙏

[noah-mfv]

My Case: I need to call many requests in synchronized order -> so, the RTT has to be high because of network latency.
I wonder if it's a problem because it definitely doesn't use much memory. Is it ok to ignore the warning?

[mperham]

@noah-mfv This doesn't make sense to me.

[noah-mfv]

Thanks for your attention! @mperham . It's my bad, I actually misunderstand the situation.

My Case: I need to call many requests in synchronized order -> so, the RTT has to be high because of network latency.

You're right, the case makes no sense. And I also found the cause of high RTT, just like what you wrote at the top page.

[collimarco]

Same Redis instance.
Now we just moved some Sidekiq workers to Kubernetes and we randomly see this warning.
On the Kubernetes nodes, the load average on each processor is at most 1.5 and RAM usage is 70% (so these metrics look good).
Maybe the containers / k8s networking is adding latency?

[nateberkopec]

See @mperham's above comment:

If your job saturates the CPU, even a concurrency of 1 can trigger the error (although it's less likely).

[dishanksimpl]

We were getting the same error, thanks to #2175 (comment) we were able to fix the issue