New parser

[ohler55]

This adds a new faster parser with:

• options localized to the individual parser,
• faster parser
• file reader/parser that uses a fixed amount of memory
• better caching (intern)

There are a few future additions that can and will be made including support for the object encoding format and possibly file reading in a separate thread.

[ohler55]

I'm still working through the CI failures but you are welcome to start looking at the MR.

[ohler55]

Okay, ready when every you want to take a look. Try the test/perf_parser.rb. Nice results there. Over 3 times faster than the current parser for compat mode.

[Watson1978]

Shouldn't create Parser instances often?
Seems that it causes memory leak.

Test code

require 'oj'

json =<<-EOF
{
  "$id": "https://example.com/person.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
}
EOF

100_000.times do |i|
  Oj::Parser.new(:usual).parse(json)

  if i % 10_000 == 0
    rss = Integer(`ps -o rss= -p #{Process.pid}`) / 1024.0
    puts "#{i},#{rss} MB"
  end
end

Result

$ ruby json.rb
0,28.390625 MB
10000,88.546875 MB
20000,150.03125 MB
30000,208.296875 MB
40000,260.734375 MB
50000,328.25 MB
60000,395.015625 MB
70000,446.171875 MB
80000,500.5625 MB
90000,547.171875 MB

[ohler55]

Normally you would resuse the parser many time. Basically set the options the way you want and use that for all parsing withing a thread.

I look for the memory leak. That should not happen.

It's rather late to Japan now, isn't it?

[ohler55]

Memory leak fixed although there seems to be a little trickle from something that I haven't identified.

[Watson1978]

Normally you would resuse the parser many time. Basically set the options the way you want and use that for all parsing withing a thread.

I see.

I'd like to use Oj.load interface to parse json in several places.
because I don't want to manage the parser instance to reuse it.

And it can migrate to new parser easily in application code.

[Watson1978]

If the object passed torb_gc_register_address() is no longer needed, you have to call rb_gc_unregister_address() to release wasted object in cache_free().

[Watson1978]

We might need to check other places that use rb_gc_register_address() as well to avoid memory leak.

[ohler55]

Good point. The cache_free will have to do that.

[ohler55]

Changed to rb_gc_mark. That seems to be faster as well in the grans scheme of things.

It was a bit surprising to note that the GC runs concurrently with the main thread. I wonder if a lock is needed for the parser to avoid a race condition with marking and parsing.

[ohler55]

Added Oj::Parser.usual which can be used like Oj::Parser.usual.parse('[true]') similar to Oj::load.

[Watson1978]

Looks good to me.