Reduce RTT readings warnings

[RachalCassity]

Discovery

Look into possible causes for the following warning.

• Ensure all redis cluster configurations are correct.
• Look into possible memory leaks

Warning

Your Redis network connection is performing extremely poorly.
Last RTT readings were [54881, 100896, 98796, 53572, 199853], ideally these should be < 1000.
Ensure Redis is running in the same AZ or datacenter as Sidekiq.
If these values are close to 100,000, that means your Sidekiq process may be
CPU-saturated; reduce your concurrency and/or see https://github.com/mperham/sidekiq/discussions/5039

Logs

https://vagov.ddog-gov.com/account/login?next=%2Flogs%3Fquery%3Denv%253Aeks-prod%2520service%253Avets-api%2520%2522CPU%2522%26cols%3Dhost%252Cservice%252Cpod_name%26event%3DAgAAAY1W2ayabC9a9gAAAAAAAAAYAAAAAEFZMVcyYkp6QUFDMjhBdllvQ3d4Q1FCRgAAACQAAAAAMDE4ZDU2ZGEtYjE1Mi00NmJmLWFhYTMtZjEyMmI2Yjk4NjFk%26index%3D%252A%26messageDisplay%3Dinline%26refresh_mode%3Dsliding%26stream_sort%3Ddesc%26viz%3Dstream%26from_ts%3D1706556427000%26to_ts%3D1706560027000%26live%3Dtrue

Sidekiq Discussions

sidekiq/sidekiq#4851

sidekiq/sidekiq#4851

[LindseySaari]

@jennb33 let's check with Reliability to make sure efforts aren't being duplicated

[jennb33]

@ericboehs is this work that Reliability team is already doing? Please advise, thanks!

[ericboehs]

RTT Redis

• CPU on pod is pegged to 50% from about 40 mins after it's created until it's terminated (~23 hours).
• 
• It appears to be the EVSS::FailedClaimsReport? JID cbd6bd6880a7acdb3610f44d and 7655b0bbb1236af4b9f727aa
  
• Looks like it's 5:
  
• Looking at what jobs are running in Sidekiq, the 5 containers with high CPU are all stuck running EVSS:FailedClaimsReport