Segmentation fault at after upgrading to Rails 6.0.0

[equivalent]

Steps to reproduce

I've upgraded application from Rails 5.2 to 6.0.0 now every time I run RSpec at the end of it I get message below

Finished in 49.25 seconds (files took 9.17 seconds to load)
25 examples, 0 failures

[BUG] Segmentation fault at 0x0000000000000050
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0001 p:---- s:0003 e:000002 (none) [FINISH]


-- Machine register context ------------------------------------------------
RIP: 0x00007f531acb47c3 RBP: 0x00007f5307576e50 RSP: 0x00007f5307576c90
RAX: 0x0000000000000000 RBX: 0x00007f531acb4a70 RCX: 0x000055d53b7b2030
RDX: 0x000055d53b7b1c18 RDI: 0x000055d5463a4848 RSI: 0x000055d5463a45f0
R8: 0x0000000000000000  R9: 0x000055d547ff6ac9 R10: 0x000055d547ff6ab0
R11: 0x00007f531b0106b0 R12: 0x000055d5463a45f0 R13: 0x00007f5307577700
R14: 0x00007f531b002a90 R15: 0x00007f5307576ea8 EFL: 0x0000000000010297

-- C level backtrace information -------------------------------------------
corrupted double-linked list
Aborted (core dumped)

funny thing is the test pass (green) but I get error message at the end

the error happens pretty much on any type of tests, nothing specific to test itself. All that was needed is Rails 6 app and RSpec

I don't know if this is Rspec error or Rails error or Ruby error 🤔

System configuration

Rails version: 6.0.0

Ruby version: 2.6.3p62

happnes:

• in my laptop (Ubuntu 18)
• in codeship VM

[eileencodes]

Thanks for the report, that's kind of surprising. We've been running Rails 6 at GitHub for awhile now and haven't seen anything like this.

This sounds like it'd be hard to reproduce and if it was simply Rails + Rspec I'd think we'd have other reports by now. Can you try reproducing in a test app? Or remove gems from your Gemfile until it goes away? Did you check Rspec issue tracker if anyone else reported something similar?

Unfortunately there's not a whole lot we can investigate with this trace and no knowledge of your app's specifics 😞

[eileencodes]

Also are you seeing this in production or just when running tests? Are you using the Rails parallelization (the process parallelizer doesn't work with Rspec btw)?

[eileencodes]

I'm going to close this for now since I haven't heard back. If you can reproduce this in a test app let me know and I'll reopen.

[davidw]

I hit this same thing.

This is when running tests.

I don't know if "rails" is at all the right project for it, since it's evidently a problem with Ruby itself, but this is the only hit with Google for that version of Rails and the 'double-linked list' message.

ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0001 p:---- s:0003 e:000002 (none) [FINISH]


-- Machine register context ------------------------------------------------
 RIP: 0x00007fe58f2246e3 RBP: 0x00007fe57bffee50 RSP: 0x00007fe57bffec90
 RAX: 0x0000000000000000 RBX: 0x00007fe58f224990 RCX: 0x000055a3aa046030
 RDX: 0x000055a3aa045c18 RDI: 0x000055a3b0986d38 RSI: 0x000055a3b07a95e0
  R8: 0x0000000000000000  R9: 0x000055a3b0e8e5e1 R10: 0x000055a3b0e8e5c8
 R11: 0x00007fe58f5806b0 R12: 0x000055a3b07a95e0 R13: 0x00007fe57bfff700
 R14: 0x00007fe58f572a90 R15: 0x00007fe57bffeea8 EFL: 0x0000000000010297

-- C level backtrace information -------------------------------------------
corrupted double-linked list

[eileencodes]

As I said to @equivalent we can't debug this without a way to reproduce it. Please provide a sample application that reproduces the segfault.

[davidw]

Probably worth it for most people to have a look at Ruby 2.6.4 - that's going to be my first step. If I still see it, I'll see what I can put together to reproduce it. It does not happen every single run, but is somewhat random.

[davidw]

Well...shoot :-(

ruby 2.6.4p104 (2019-08-28 revision 67798) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0001 p:---- s:0003 e:000002 (none) [FINISH]


-- Machine register context ------------------------------------------------
 RIP: 0x00007fda8c362753 RBP: 0x00007fda7d203e50 RSP: 0x00007fda7d203c90
 RAX: 0x0000000000000000 RBX: 0x00007fda8c362a00 RCX: 0x000055ad88706030
 RDX: 0x000055ad88705c18 RDI: 0x000055ad8f77d088 RSI: 0x000055ad8f77ce30
  R8: 0x0000000000000000  R9: 0x000055ad8f59bef1 R10: 0x000055ad8f59bed8
 R11: 0x00007fda8c6c0610 R12: 0x000055ad8f77ce30 R13: 0x00007fda7d204700
 R14: 0x00007fda8c6b2a10 R15: 0x00007fda7d203ea8 EFL: 0x0000000000010297

-- C level backtrace information -------------------------------------------
corrupted double-linked list
Aborted (core dumped)

[davidw]

I've started narrowing things down to something funny with mongoid and database_cleaner. @equivalent does that ring a bell at all?

Almost positive it has little or nothing to do with Rails itself; I'll try and leave a final note for anyone searching via Google if I figure it out.

[amatsuda]

If I'm seeing segfaults on my Rails app, I would try this first.

diff --git a/Gemfile b/Gemfile
index 28a671a..2c3e3eb 100644
--- a/Gemfile
+++ b/Gemfile
@@ -25,9 +25,6 @@ gem 'jbuilder', '~> 2.7'
 # Use Active Storage variant
 # gem 'image_processing', '~> 1.2'
 
-# Reduces boot times through caching; required in config/boot.rb
-gem 'bootsnap', '>= 1.4.2', require: false
-
 group :development, :test do
   # Call 'byebug' anywhere in the code to stop execution and get a debugger console
   gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
diff --git a/config/boot.rb b/config/boot.rb
index b9e460c..30f5120 100644
--- a/config/boot.rb
+++ b/config/boot.rb
@@ -1,4 +1,3 @@
 ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)
 
 require 'bundler/setup' # Set up gems listed in the Gemfile.
-require 'bootsnap/setup' # Speed up boot time by caching expensive operations.

[davidw]

Ok, I have a somewhat self-contained Rails app that causes the crash. But it's definitely something to do with mongo/mongoid and database_cleaner. Removing this makes things work:

config.after(:each) do
    DatabaseCleaner[:mongoid].clean
  end

Trying to narrow things down further.

@amatsuda I have removed that gem too, among many others, and it still crashes.

[davidw]

I am 99% sure the bug is this:

ruby-concurrency/concurrent-ruby#808

[vitalinfo]

@davidw I have the same problem on Ruby 2.6.3 and 2.6.4
I've commented DatabaseCleaner[:mongoid].clean result the same

[eileencodes]

OK sounds like this isn't a Rails issue, can you take this up with the appropriate projects and followup when it's fixed? Thanks!

[stefanjohncollier]

Would you mind posting those new issues no here so that we can follow along please? 🙏

[davidw]

@stefanjohncollier as far as I can tell, the issue is the Ruby Concurrency one I linked to: ruby-concurrency/concurrent-ruby#808

Although of course I think there must be a bug in the Ruby interpreter as well if it's seg faulting.

[stefanjohncollier]

Ah thanks @davidw, I'll give that a subscribe!

[equivalent]

@davidw my apology I forgot to reply to your comment yes I'm using mongoid and Database cleaner too same way as you are 🤔

I'll try play around with disabling some parts of mongo tests in my app to see what happens and let you know. But if it's really Ruby side (link) then that will be fun one to find 😢

@eileencodes sorry I didn't reply to your initial comment but I wasn't able to pinpoint anything how to reproduce this at the time.

Thank you @davidw for doing the hard work digging deeper around this issue 🙏

---

update later that day: nope tried to disable bunch of mongoid / database cleaner related tests or configurations , upgraded mongoid ref didn't work 🤔

[chaadow]

@eileencodes
I'm having segfault errors when running tests against the master branch of rails on MacOS ( with parallel tests on), and it's random, so can't consistently reproduce it.

idk if i should open an issue or not, I've also got the crash report just in case you think it's ok to open an issue.

        7: from -e:1:in `<main>'
        6: from /Users/xx/.rbenv/versions/2.6.5/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        5: from /Users/xx/.rbenv/versions/2.6.5/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        4: from /Users/xx/.rbenv/versions/2.6.5/gemsets/xxx/bundler/gems/rails-d2e8b839d2e7/activesupport/lib/active_support/fork_tracker.rb:8:in `fork'
        3: from /Users/xx/.rbenv/versions/2.6.5/gemsets/xxx/bundler/gems/rails-d2e8b839d2e7/activesupport/lib/active_support/fork_tracker.rb:8:in `fork'
        2: from /Users/xx/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.12.2/lib/minitest.rb:66:in `block in autorun'
        1: from /Users/xx/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.12.2/lib/minitest.rb:143:in `run'

       /Users/username/.rbenv/versions/2.6.5/gemsets/some_gemset_name/bundler/gems/rails-d2e8b839d2e7/activesupport/lib/active_support/testing/parallelization.rb:129:in `shutdown': 

**Queue not empty, but all workers have finished. This probably means that a worker crashed and 1 tests were missed. (RuntimeError)**

( it's coming from the /activesupport/lib/active_support/testing/parallelization file )

[eileencodes]

Please provide a test app with only Rails gems that reproduces the issue. Without that the Rails team cannot debug the problem.

[davidw]

For anyone else running into this, I have filed a bug against Ruby itself, as segfaults should not happen no matter what the Ruby code is doing wrong:

https://bugs.ruby-lang.org/issues/16288

It includes (in my latest comment) code that reliably segfaults Ruby without Rails, Mongo, concurrent-ruby or any of that.

[equivalent]

solution here ruby-concurrency/concurrent-ruby#808 (comment)