Add pumactl command to print thread backtraces #2054

composerinteralia · 2019-10-25T03:01:25Z

Completes 1 of 2 items from #1964

This commit adds an endpoint to the status app to print thread
backtraces, and control cli command to call that endpoint.

I tried this locally by starting a server with:

bundle exec bin/puma test/rackup/hello.ru \
  --control-url="unix://test.sock" \
  --control-token="token"

and then printing the backtraces with:

bundle exec bin/pumactl thread-backtraces \
  --control-url="unix://test.sock" \
  --control-token="token"

I have reviewed the guidelines for contributing to this repository.
I have added an entry to History.md if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added [changelog skip] to all commit messages.
I have added appropriate tests if this PR fixes a bug or adds a feature.
My pull request is 100 lines added/removed or less so that it can be easily reviewed.
If this PR doesn't need tests (docs change), I added [ci skip] to the commit messages.
If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
I have updated the documentation accordingly.
All new and existing tests passed, including Rubocop.

composerinteralia · 2019-10-26T02:26:23Z

lib/puma/app/status.rb

+        when /\/thread-backtraces$/
+          strings = Puma::Events.strings
+          @cli.log_thread_status(strings)
+          rack_response(200, strings.stdout.string)


Should this be JSON instead of the raw string?

Yes. All current responses are json, so we're going to need to turn this into json object.

Maybe something like:

threads: [ { name: "", backtrace: "" }, { name: "", backtrace: "" } ]

nateberkopec · 2019-10-31T13:26:39Z

lib/puma/app/status.rb

+        when /\/thread-backtraces$/
+          strings = Puma::Events.strings
+          @cli.log_thread_status(strings)
+          rack_response(200, strings.stdout.string)


Yes. All current responses are json, so we're going to need to turn this into json object.

Maybe something like:

threads: [ { name: "", backtrace: "" }, { name: "", backtrace: "" } ]

nateberkopec · 2019-10-31T13:26:50Z

lib/puma/control_cli.rb

@@ -11,7 +11,8 @@
 module Puma
  class ControlCLI

-    COMMANDS = %w{halt restart phased-restart start stats status stop reload-worker-directory gc gc-stats}
+    COMMANDS = %w{halt restart phased-restart start stats status stop reload-worker-directory gc gc-stats thread-backtraces}
+    PRINTABLE_COMMANDS = %w{gc-stats stats thread-backtraces}


nateberkopec · 2019-10-31T13:27:58Z

lib/puma/launcher.rb

@@ -205,6 +205,21 @@ def close_binder_listeners
      @binder.close_listeners
    end

+    def log_thread_status(events)


Looks like you'll need to refactor this out a bit so that you have two methods - one that prepares a Ruby object representation of threads and backtraces, then one method that logs that info to events.

@nateberkopec Should we always use the object representation, or do you want to preserve the existing format for the SIGINFO and use the object representation only for the status app?

I think what matters is that the current logged output remains the same, the status app outputs JSON, and that they both use the same method underneath for obtaining that info.

Completes 1 of 2 items from puma#1964 This commit adds an endpoint to the status app to print thread backtraces, and control cli command to call that endpoint. I tried this locally by starting a server with: ```sh bundle exec bin/puma test/rackup/hello.ru \ --control-url="unix://test.sock" \ --control-token="token" ``` and then printing the backtraces with: ```sh bundle exec bin/pumactl thread-backtraces \ --control-url="unix://test.sock" \ --control-token="token" ```

With this commit the status app sends the thread backtraces as an array of objects with `name` and `backtrace` keys, rather than as a string matching the SIGINFO output. While working on this I noticed that we logged the thread TID twice. This commit simplifies that so we only log the thread TID once, with both the label (I don't know when the label would get set) and name if they are available.

composerinteralia · 2019-11-11T04:14:00Z

Ready for another review

nateberkopec · 2019-11-11T05:08:52Z

Nice work!

composerinteralia · 2019-11-11T05:09:40Z

Woah, that was fast! Thanks!

olleolleolle · 2019-11-11T06:22:53Z

From the sidelines: this is super helpful! The tab that was added to each backtrace line makes for an ergonomic developer experience.

jondavidschober · 2020-04-01T21:45:03Z

Is there a timeline for releasing this?

nateberkopec · 2020-04-01T22:04:17Z

This was merged 3 days after 4.3 was released, so I realize it's been in the backlog for a while. You can see progress on 5.0 here: https://github.com/puma/puma/milestone/7

composerinteralia force-pushed the pumactrl-thread-backtraces branch 3 times, most recently from a52fd51 to 7eb7ad8 Compare October 26, 2019 01:46

composerinteralia marked this pull request as ready for review October 26, 2019 01:50

composerinteralia commented Oct 26, 2019

View reviewed changes

nateberkopec added the feature label Oct 26, 2019

nateberkopec requested changes Oct 31, 2019

View reviewed changes

composerinteralia force-pushed the pumactrl-thread-backtraces branch from 7eb7ad8 to e87d82d Compare November 11, 2019 03:52

composerinteralia force-pushed the pumactrl-thread-backtraces branch from e87d82d to a344d75 Compare November 11, 2019 03:58

nateberkopec merged commit 39d16fa into puma:master Nov 11, 2019

composerinteralia deleted the pumactrl-thread-backtraces branch November 11, 2019 05:09

nateberkopec mentioned this pull request Jun 8, 2020

PUMA stops responding after a few minutes idle time. LSOF -wni tcp:port shows CLOSE_WAIT connections #2294

Closed

nateberkopec mentioned this pull request Oct 1, 2020

Ctrl+C takes a long time to quit in 5.0.x #2398

Closed

dentarg mentioned this pull request Oct 6, 2020

Feature idea: puma config command #2414

Closed

cjlarose mentioned this pull request Oct 30, 2020

phased-restart errors after upgrading from Puma 3.12.6 to 5.0.4 #2471

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pumactl command to print thread backtraces #2054

Add pumactl command to print thread backtraces #2054

composerinteralia commented Oct 25, 2019 •

edited

composerinteralia Oct 26, 2019

nateberkopec Oct 31, 2019

nateberkopec Oct 31, 2019

nateberkopec Oct 31, 2019

nateberkopec Oct 31, 2019

composerinteralia Nov 8, 2019

nateberkopec Nov 9, 2019

composerinteralia commented Nov 11, 2019

nateberkopec commented Nov 11, 2019

composerinteralia commented Nov 11, 2019

olleolleolle commented Nov 11, 2019

jondavidschober commented Apr 1, 2020

nateberkopec commented Apr 1, 2020

Add pumactl command to print thread backtraces #2054

Add pumactl command to print thread backtraces #2054

Conversation

composerinteralia commented Oct 25, 2019 • edited

composerinteralia Oct 26, 2019

Choose a reason for hiding this comment

nateberkopec Oct 31, 2019

Choose a reason for hiding this comment

nateberkopec Oct 31, 2019

Choose a reason for hiding this comment

nateberkopec Oct 31, 2019

Choose a reason for hiding this comment

nateberkopec Oct 31, 2019

Choose a reason for hiding this comment

composerinteralia Nov 8, 2019

Choose a reason for hiding this comment

nateberkopec Nov 9, 2019

Choose a reason for hiding this comment

composerinteralia commented Nov 11, 2019

nateberkopec commented Nov 11, 2019

composerinteralia commented Nov 11, 2019

olleolleolle commented Nov 11, 2019

jondavidschober commented Apr 1, 2020

nateberkopec commented Apr 1, 2020

composerinteralia commented Oct 25, 2019 •

edited