Taking a Stack Dump

If things seem to be "stuck", there may be a smoking gun available in the form of a Goroutine stack dump. This will reveal goroutines that are stuck in suspicious states like IO wait and how long they've been stuck in that state.

Via the Debug Server

This can be acquired by hopping on the machine running the web node and curling the following endpoint:

Web

http://127.0.0.1:8079/debug/pprof/goroutine?debug=2

Worker

http://127.0.0.1:7776/debug/pprof/goroutine?debug=2

This will spit out a ton of text, so you may want to redirect its output to a file (or just copy-paste it all from your terminal output).

Note that this has to be done from the same machine because the debug endpoint only listens on the loopback 127.0.0.1 address.

Via SIGQUIT

Any Go process can be sent SIGQUIT, which will send this very same stack dump to stderr. The downside of this method is that it kills the process.

This can be done like so:

kill -QUIT $pid

...where $pid should be replaced with whatever the pid is from ps.

Note that you'll have to look wherever stderr is being redirected in order to collect the stack dump.

Analyzing Stack Dump

Once dump is ready then you could put it in clipboard and paste it in swirly for a visualized view of the dump.