timeout presence leads to MiniRacer::ScriptTerminatedError instead of MiniRacer::V8OutOfMemoryError

[thepry]

UPD: I can confirm that gc_callback can get executed after rb_context_eval_unsafe was called, but before nogvl_context_eval. MEM_SOFTLIMIT_REACHED is set to false in nogvl_context_eval, but isolate execution would still get terminated, leading to the wrong exception being raised.

It looks like some operations inside of timeout function(https://github.com/rubyjs/mini_racer/blob/master/lib/mini_racer.rb#L328), like Mutex.new are just slow enough to allow for this race condition to happen in the test example.

---

I've noticed a weird behavior when I set the timeout for the context, it sometimes throws MiniRacer::ScriptTerminatedError exception instead of expected MiniRacer::V8OutOfMemoryError. The script executes fast enough so there shouldn't be the actual timeout error.

Here is an example tests that reproduced the issue:

  # This will pass
  def test_max_memory_2
    context = MiniRacer::Context.new(max_memory: 1_000_000)

    assert_raises(MiniRacer::V8OutOfMemoryError) do
      500.times do |i|
        script = "var t#{i} = '#{SecureRandom.uuid * 1000}'; Array.from(new Array(10000)).map((e, i) => i + 1)[1];"
        context.eval(script)
        sleep 0.001
      end
    end
  end

  # This will fail:
  def test_max_memory_3
    context = MiniRacer::Context.new(max_memory: 1_000_000, timeout: 20_000)

    assert_raises(MiniRacer::V8OutOfMemoryError) do
      500.times do |i|
        script = "var t#{i} = '#{SecureRandom.uuid * 1000}'; Array.from(new Array(10000)).map((e, i) => i + 1)[1];"
        context.eval(script)
        sleep 0.001
      end
    end
  end

.F

Failure:
MiniRacerTest#test_max_memory_3 [.../.../mini_racer/test/mini_racer_test.rb:335]:
[MiniRacer::V8OutOfMemoryError] exception expected, not
Class: <MiniRacer::ScriptTerminatedError>
Message: <"JavaScript was terminated (either by timeout or explicitly)">
---Backtrace---
.../.../mini_racer/lib/mini_racer.rb:209:in `eval_unsafe'
.../.../mini_racer/lib/mini_racer.rb:209:in `block (2 levels) in eval'
.../.../mini_racer/lib/mini_racer.rb:350:in `timeout'
.../.../mini_racer/lib/mini_racer.rb:208:in `block in eval'
.../.../mini_racer/lib/mini_racer.rb:206:in `synchronize'
.../.../mini_racer/lib/mini_racer.rb:206:in `eval'
.../.../mini_racer/test/mini_racer_test.rb:338:in `block (2 levels) in test_max_memory_3'
.../.../mini_racer/test/mini_racer_test.rb:336:in `times'
.../.../mini_racer/test/mini_racer_test.rb:336:in `block in test_max_memory_3'

I suspect that might be some sort of race condition with V8 GC and MEM_SOFTLIMIT_REACHED flag (https://github.com/rubyjs/mini_racer/blob/master/ext/mini_racer_extension/mini_racer_extension.cc#L336, https://github.com/rubyjs/mini_racer/blob/master/ext/mini_racer_extension/mini_racer_extension.cc#L217), but I haven't got a chance to dig into that yet.

ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-darwin19]
VERSION = "0.3.1"
LIBV8_VERSION = "~> 8.4.255"
OSX