Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging SignalException when gracefully shutting down Puma on Heroku #267

Closed
calleluks opened this issue Feb 23, 2018 · 5 comments
Closed
Assignees

Comments

@calleluks
Copy link

Hello!

I'm monitoring a Rails 5.1.5 app running on Heroku with Honeybadger using version 3.3.0 of the gem.

The app uses the default Rails puma server configuration.

For some time now, Honeybadger has been logging a SignalException every time I deploy the application. Here's an example of what the Heroku log output looks like when the app restarts after a deploy:

Feb 23 14:57:42 heroku/web.1:  Restarting 
Feb 23 14:57:42 heroku/web.1:  State changed from up to starting 
Feb 23 14:57:43 heroku/web.1:  Stopping all processes with SIGTERM 
Feb 23 14:57:43 app/web.1:  - Gracefully stopping, waiting for requests to finish 
Feb 23 14:57:43 app/web.1:  === puma shutdown: 2018-02-23 13:57:42 +0000 === 
Feb 23 14:57:43 app/web.1:  - Goodbye! 
Feb 23 14:57:43 app/web.1:  Exiting 
Feb 23 14:57:43 app/web.1:  I, [2018-02-23T13:57:42.976150 #4]  INFO -- : ** [Honeybadger] Reporting error id=REDACTED level=1 pid=4 
Feb 23 14:57:44 heroku/web.1:  Process exited with status 143 
Feb 23 14:57:44 app/web.1:  I, [2018-02-23T13:57:43.342196 #4]  INFO -- : ** [Honeybadger] Success ⚡ https://app.honeybadger.io/notice/REDACTED id=REDACTED code=201 level=1 pid=4 

I think this might be related to this change in Puma which aims to properly handle the TERM signal by re-raising it and letting Ruby's built in handling exit with the correct status code:

https://github.com/puma/puma/blob/dc9fa77f855c5018a03430366ea8c8db17fbfeea/lib/puma/launcher.rb#L397

~$ irb
irb(main):001:0> raise SignalException, 'SIGTERM'
Terminated: 15
~$ echo $?
143

Having this exception logged every time I deploy is not what I want but I'm not sure what needs to be fixed here. Should Honeybadger not rescue this exception? Should I ignore it manually in my configuration? Would I then risk missing a legitimate application error?

@minhajuddin
Copy link
Contributor

This is an interesting situation, let me look at our code and see what is possible.

@calleluks
Copy link
Author

Thanks, @minhajuddin!

@ckersh
Copy link

ckersh commented Mar 9, 2018

I am seeing a similar issue using Rails 5.1.4, Passenger 5.2.1, and Honeybadger 3.2.0.

@minhajuddin
Copy link
Contributor

In our at_exit callback (

if $! && !$!.is_a?(SystemExit) && Honeybadger.config[:'exceptions.notify_at_exit']
) we ignore SystemExit exceptions, adding another clause for SignalException caused by a SIGTERM should fix this issue.

SIGTERM is usually sent by admins when they want to kill processes and not by the kernel. So, it should be safe for us to ignore this. Thoughts? @joshuap

@joshuap
Copy link
Member

joshuap commented Mar 20, 2018

I think that would be OK, @minhajuddin 👍

patbl added a commit to patbl/Skillshare.im that referenced this issue Mar 21, 2020
This'll prevent reporting of "SignalException in at_exit" errors:
honeybadger-io/honeybadger-ruby#267
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants