Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passenger workers hang / segfault with Ruby 2.5 / concurrent-ruby 1.1.6 #871

Open
astupka opened this issue May 15, 2020 · 3 comments
Open
Labels
bug A bug in the library or documentation.

Comments

@astupka
Copy link

astupka commented May 15, 2020

Thank you for all your hard work on the latest 1.1.6 release.

I recently upgraded our app gems and started noticing the passenger workers would become unresponsive (request queue = 100) and not shutdown (normal mem/no CPU usage) after ~12-24 hours. Inspecting the error logs, I found some segfaults that appeared related to the finalization code present in 1.1.6 and a possible MRI bug. Rolling back this single gem to 1.1.5 resolved the issue.

Phusion Passenger 5.0.24
Rails 6.0.0
* Operating system:                Ubuntu 14.04.6 LTS
* Ruby implementation:             ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
* `concurrent-ruby` version:      1.1.6
* `concurrent-ruby-ext` installed: no
* `concurrent-ruby-edge` used:     no

Initial segfault data (let me know if the rest would be helpful)

App 26150 stderr: /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thread_local_var.rb:113:
App 26150 stderr: [BUG]
App 26150 stderr: Segmentation fault at 0x0000000000000050
App 26150 stderr:
App 26150 stderr: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
App 26150 stderr:
App 26150 stderr: -- Control frame information -----------------------------------------------
App 26150 stderr: c:0003
App 26150 stderr: p:----
App 26150 stderr: s:0011
App 26150 stderr: e:000010
App 26150 stderr: CFUNC
App 26150 stderr:  :push
App 26150 stderr:
App 26150 stderr: c:0002
App 26150 stderr: p:0017
App 26150 stderr: s:0006
App 26150 stderr: e:000005
App 26150 stderr: BLOCK
App 26150 stderr:  /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thr
App 26150 stderr:  [FINISH]
App 26150 stderr:
App 26150 stderr: c:0001
App 26150 stderr: p:0000
App 26150 stderr: s:0003
App 26150 stderr: E:0022e0
App 26150 stderr: (none)
App 26150 stderr:  [FINISH]
App 26150 stderr:
App 26150 stderr:
App 26150 stderr: -- Ruby level backtrace information ----------------------------------------
App 26150 stderr: /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thread_local_var.rb:113:in `block in thread_local_finalizer'
App 26150 stderr: /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thread_local_var.rb:113:in `push'
App 26150 stderr:
App 26150 stderr: -- Machine register context ------------------------------------------------
App 26150 stderr:  RIP: 0x00007f3d1c008d96
App 26150 stderr:  RBP: 0x0000000000000000
App 26150 stderr:  RSP: 0x00007ffed1b32bb0
App 26150 stderr:
App 26150 stderr:  RAX: 0x00007f3d187d61f8
App 26150 stderr:  RBX: 0x00007f3d1c0084e4
App 26150 stderr:  RCX: 0x00007f3d1e742ba8
App 26150 stderr:
App 26150 stderr:  RDX: 0x00007f3d1c0084ec
App 26150 stderr:  RDI: 0x0000000000000000
App 26150 stderr:  RSI: 0x00007f3d21a0c8e0
App 26150 stderr:
App 26150 stderr:   R8: 0x00007f3d1c2bc068
App 26150 stderr:   R9: 0x00007f3d1ddd5b20
App 26150 stderr:  R10: 0x0000000000000001
App 26150 stderr:
App 26150 stderr:  R11: 0x00007f3d14a0e199
App 26150 stderr:  R12: 0x00007f3d21a2f7a0
App 26150 stderr:  R13: 0x00007f3d232ac4d0
App 26150 stderr:
App 26150 stderr:  R14: 0x00007f3d1dd06248
App 26150 stderr:  R15: 0x00007f3d21a2f7a0
App 26150 stderr:  EFL: 0x0000000000010287
App 26150 stderr:
App 26150 stderr:
@astupka astupka changed the title Passenger worker hangs / segfaults with Ruby 2.5 / concurrent-ruby 1.1.6 Passenger workers hang / segfault with Ruby 2.5 / concurrent-ruby 1.1.6 May 15, 2020
@astupka
Copy link
Author

astupka commented May 15, 2020

Could be addressed in this PR #856

@crevete
Copy link

crevete commented Jun 18, 2020

I've encountered the same issue with ruby 2.6.6, rails 5.2.4.3, and only when running unit tests with fork call.

[BUG] Segmentation fault at 0x00007f685c9559d0
ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]

When rolling back this gem to 1.1.5, the Segmentation Fault disappears.

UPDATE: with 1.1.5 the Segmentation Fault may still be there...

@pitr-ch pitr-ch added the bug A bug in the library or documentation. label Jul 20, 2020
@pitr-ch
Copy link
Member

pitr-ch commented Jul 20, 2020

#856 was merged plese test with master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A bug in the library or documentation.
Projects
None yet
Development

No branches or pull requests

3 participants