Autoloader racing on multiple threads #3040
Comments
|
I think you are seeing #2431. Thanks for the detailed issue and reproduction. |
|
Didn't get too far, still got confused by the nodes layout in execution. I've used the following code for debugging (safe enough to avoid racing but still threaded so the same-thread-locking guards can be tested): require("thread")
autoload("Foo", "./foo.rb")
Foo.new.bar
Thread.new { Foo.new.bar }.joinAt the first GetConstantNode again (couldn't figure out why it needs a second one). This eventually replaces the constant entry for Object:
. And now this can return to the parent GetConstantNode and resolve the autoload lock: Now when we go to the second lookup for Only had vague ideas how to approach it:
I'm looking for some guidance on a better approach here. |
|
One idea is for the constant being autoloaded to not set it yet but either store it in a new field of AutoloadConstant or maybe in the RubyConstant#value of the autoload RubyConstant. |
|
A reliable reproducer: # main.rb
autoload :C, "#{__dir__}/c.rb"
$inside = false
t = Thread.new { p C.new.foo }
Thread.pass until $inside
p C.new.foo
t.join# c.rb
class C
def bar
end
$inside = true
sleep 1
def foo
42
end
end |
|
I tried an edge case, what if while loading the file we create another thread which tries to access the being-autoloaded constant? Answer: it deadlocks on CRuby: # c.rb
class C
def bar
end
Thread.new { p C }.join
$inside = true
sleep 1
def foo
42
end
endSo it's simple, we don't need to handle that case, it can just hang. If we simplify main.rb we can get CRuby to print the deadlock: # main.rb
autoload :C, "#{__dir__}/c.rb"
p C.new.foogives |
|
With Fibers on 3.2, with So just an early error but still a deadlock. |
|
I have a fix for this in #3078, I'm still polishing it a bit. I also noticed we still have the notion of "undefined constants", but this should have been removed in CRuby 3.1, so now we should be able to clean that up as well. But probably best done separately, it's already quite a complex change as-is. |
eregon
modified the milestones:
23.1.0 Release (September 19, 2023),
24.0.0 Release (March 19, 2024)
eregon
modified the milestones:
24.0.0 Release (March 19, 2024),
24.1.0 Release (Sep 17, 2024)
Autoloader use with multiple threads cause premature use of the class.
Example
When executing mutliple
Railstests with the debugger:jt --use jvm-ce ruby --jdebug -S ./bin/rails test test/folderthe execution fails with:The test executor spawns threads to spread the test execution, and most threads fail with this error (not all). What's happening is the very first thread is referring to
ActiveSupport::CurrentAttributes, an unknown constant. The autoload definition in https://github.com/rails/rails/blob/2bfb6565a4996a19065e03300d9061e9eb265d92/activesupport/lib/active_support.rb#L42 delegates the resolution to Zeitwerk and it starts loading the class (snippet from the first thread load backtrace):Meanwhile the other threads will just use the class as if it's already loaded - which is not the case, the class's Ruby code is still executing (function definitions) so some threads won't have all the methods available: hence the error. This was confirmed by logging file execution start/end.
Reproduce
rails generate scaffold PostA name:string title:string content:text value:integer--jdebugMinimum example
Unfortunately reproducing it with a minimum example was tricky. It seems using
Module#autoladvsActiveSupport::Autoload#autoloadwas the same. I couldn't replicate it with--jvm-cebut saw something similar with--native.Code
Example output (6 failure, 2 success):
Observation
--jvm-cedoes not reproduce the error with the snippet above--jdebugon the Rails test run because without it there is a different racing error (tracked separately), probably the debugger slows it down enough so the first error does not occur that often--engine.Compilation=false)The text was updated successfully, but these errors were encountered: