Support a "debug" mode

[mperham]

It is not uncommon for people to get connection pool timeouts because all connections have been checked out. Connection leakage happens and being able to track down which Thread checked them out and the corresponding backtrace would be a useful feature.

Initial questions

• Are there other connection pool libraries which do something similar that we can borrow ideas from? Does AR do anything like this?
• What info do we collect?
• How do we enable debug mode?
• How do we trigger the info dump?
• Where do we write the info dump?
• We might have many connection pools in a process, how do we target a specific one?

Initial ideas

Imagine defining a connection_pool with a unique name, like this:

conn = ConnectionPool.new(size: 5, timeout: 5, name: 'sidekiq_main')

If debug mode is enabled, every connection checkout would collect the backtrace and current thread so we can output it in the future.

We could define an environment variable which enables debug mode:

CONNECTION_POOL_DEBUG=sidekiq_main:TTIN:/tmp/sidekiq_main.txt bundle exec sidekiq
CONNECTION_POOL_DEBUG=sidekiq_main:USR2:/dev/stdout bundle exec sidekiq
CONNECTION_POOL_DEBUG=${name}:${signal}:${output}

This would install a ${signal} hander for the ${name} connection pool that logs debug info to ${output} upon receiving that signal. I like having the ability to debug without changing the code at all. You would need to restart the process to enable debugging because there is significant overhead in gathering and tracking the connection and thread locations/backtraces. This is not something you'd want to enable 24/7.

Strict mode?

What about a strict mode which warns if a connection was checked out for more than N seconds? I don't like this as much. The lightweight impl would warn upon checkin which wouldn't catch leaked connections. The heavyweight impl would require a separate Thread which constantly scans for connections over the time limit; that's a heavy cost for a single pool.

Other ideas welcome.

[openbl]

This sounds like a useful tool for folks to better understand the state of their connection pools.

I was also wondering if supporting some sort of metrics/statsd/notification bus similar to Rails' ActiveSupport::Notifications so folks to tie the state of their pool into whatever system they have. A logger sink/subscriber could be the default or a configurable option especially if it's done in a plugin-like way.

Regardless, thank you for this library and the great work and support!

[jpriollaud]

thanks @mperham This helped alot, maybe some ideas:
https://tejasbubane.github.io/posts/2020-04-22-redis-connection-pool-in-rails/