debugging deadlocked threads?

[matti]

I have a rails app which I'm able to deadlock by making requests when puma is starting the workers/threads.

The only way it never deadlocks is by having:

workers 1
threads 1, 1

But when it is set to:

workers 1
threads 4, 16

Then most of the time from pumactl stats I can see that the number of running threads goes up and pool_capacity goes down to 1. With curl -v I can see that connection is opened, but no data is received and nothing is seen in the rails app logs.

How do I debug further?

[matti]

I'm running rails 6 and 2.6.3 ruby. Found #1744 (comment) - so if for example ruby 2.6.3 had some issues, how would I be able to confirm this?

[nateberkopec]

See #1865 (comment)

[matti]

@nateberkopec hmm, I don't have any log lines, so not any ThreadError (deadlock; recursive locking) messages etc.

I did however insert the middleware and http://localhost:3000/rails/locks behaves like the app - no response, the request just hangs.

I'm suspecting that there might be something in Ruby/Puma that causes this, because it only triggers on "every 3rd try".

The app sits behind nginx and I'm running this kind of request generation while booting the app, it seems to make the hang happen more often:

while true; do
   curl localhost:3000 &
   sleep 0.01
done

[matti]

I'm trying to create a repro with a new rails app, but any pointers how I could debug further in my existing app would be great!

[nateberkopec]

Are you using SSL?

[matti]

Nope - everything is http. I am running this in minikube so it is behind ingress-nginx, but no ssl/https involved. And with curl verbose mode I can see that it opens the http connection, but writes no data - and no lines in rails log after listening 0.0.0.0:3000

and it never hangs with threads 1,1 and sometimes does not hang with threads 4,16 so that makes me suspect that ruby/puma has some deadlock and not rails.

[nateberkopec]

and it never hangs with threads 1,1 and sometimes does not hang with threads 4,16 so that makes me suspect that ruby/puma has some deadlock and not rails.

That's not necessarily true. You probably just have a deadlock in your app that isn't revealed unless you're running multi-threaded.

[matti]

Yeah you are right - I just can't find any clues anywhere (no log messages etc)

Can I somehow see Puma's threads what they are doing? Like with some software you can send -TTIN signal and get debug output?

[nateberkopec]

Try applying #1320