method `session_new_cb' called on hidden T_DATA object (NotImplementedError)

[casperisfine]

A weird error that happened once in production.

Ruby 3.0.2 (linux)

Full backtrace:

/usr/local/ruby/lib/ruby/3.0.0/openssl/buffering.rb:205:in `sysread_nonblock': hidden object cannot have instance variables (TypeError)
    from /usr/local/ruby/lib/ruby/3.0.0/openssl/buffering.rb:205:in `read_nonblock'
    from gems/net-protocol-0.1.1/lib/net/protocol.rb:212:in `rbuf_fill'
    from gems/net-protocol-0.1.1/lib/net/protocol.rb:193:in `readuntil'
    from gems/net-protocol-0.1.1/lib/net/protocol.rb:203:in `readline'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http/response.rb:42:in `read_status_line'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http/response.rb:31:in `read_new'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http.rb:1557:in `block in transport_request'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http.rb:1548:in `catch'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http.rb:1548:in `transport_request'
...

Caused by: /usr/local/ruby/lib/ruby/3.0.0/openssl/buffering.rb:205:in `sysread_nonblock': method `session_new_cb' called on hidden T_DATA object (0x00007f17df28b0d8 flags=0xc) (NotImplementedError)
    from /usr/local/ruby/lib/ruby/3.0.0/openssl/buffering.rb:205:in `read_nonblock'
    from gems/net-protocol-0.1.1/lib/net/protocol.rb:212:in `rbuf_fill'
    from gems/net-protocol-0.1.1/lib/net/protocol.rb:193:in `readuntil'
    from gems/net-protocol-0.1.1/lib/net/protocol.rb:203:in `readline'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http/response.rb:42:in `read_status_line'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http/response.rb:31:in `read_new'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http.rb:1557:in `block in transport_request'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http.rb:1548:in `catch'
    from /usr/local/ruby/lib/ruby/3.0.0/net/http.rb:1548:in `transport_request'
...

We do call GC.compact once, so maybe there's a static VALUE reference that isn't properly declared to the GC?

[casperisfine]

I tried to reproduce with GC.verify_compaction_references, but it doesn't reveal anything: casperisfine@8bdb729

So maybe it's a red herring and this bug has nothing to do with GC compaction?

[rhenium]

Caused by: /usr/local/ruby/lib/ruby/3.0.0/openssl/buffering.rb:205:in sysread_nonblock': method session_new_cb' called on hidden T_DATA object (0x00007f17df28b0d8 flags=0xc) (NotImplementedError)

I believe this is coming from

openssl/ext/openssl/ossl_ssl.c

Line 398 in e3a4093

cb = rb_funcall(ssl_obj, rb_intern("session_new_cb"), 0);

The receiver is an OpenSSL::SSL::SSLSocket obtained from the OpenSSL type SSL; each SSLSocket wraps an SSL and the SSL has a reverse reference to the SSLSocket object.

Unfortunately I know little about the compactor. Do we need to mark/pin the T_DATA/SSLSocket itself explicitly in such a case?

[casperisfine]

Unfortunately I know little about the compactor.

That was just my assumption as I did deal with several compactor issues in native gems, and they usually revolve around methods being called on random objects, this one might be entirely different.

[rhenium]

This causes segfault, so this indeed seems to be the root cause.

require "socket"
require "openssl"

p1, p2 = UNIXSocket.pair
ctx = OpenSSL::SSL::SSLContext.new.tap { |x| x.security_level = 0; x.ciphers = "aNULL" }
s1 = OpenSSL::SSL::SSLSocket.new(p1, ctx)
s2 = OpenSSL::SSL::SSLSocket.new(p2, ctx)
GC.verify_compaction_references
[
  Thread.new { s1.accept },
  Thread.new { s2.connect },
].map(&:join)

[casperisfine]

Yes, @XrXr came to the same conclusion: XrXr@ad02bb9

[rhenium]

#465 should fix this by adding rb_gc_mark() calls.

[rhenium]

openssl 2.1.3 and 2.2.1 have been released with the fix. Thanks for reporting this issue!