aws-sdk-core implicitly depends on Oj gem

[leosurwiki]

According to

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/json.rb

Line 41 in a99d0ab

require 'oj'

aws-core-sdk will try to use OJ if its installed, and the version of this dependency is not controlled at all. When error raised from Oj, it also tries to convert Oj::ParseError to Aws::Json::ParseError

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/json.rb

Line 26 in a99d0ab

rescue *ENGINE_ERRORS => e

ohler55/oj#584
However, according to the Author of Oj, Oj is supposed to raise Json::ParserError, and raising Oj::ParserError is the behavior caused by a bug in a previous version.

Context:
Same issue from #2085 but still unfixed for us since we use a different Oj version.

Suggested changes could be:

• also rescue JSON::ParserError even if Oj is installed
• declare the Oj dependency explicitly
  ...

[mullermp]

FYI - We control the version of OJ depending on the Ruby version: https://github.com/aws/aws-sdk-ruby/blob/master/Gemfile#L47-L56

[mullermp]

We will investigate this.

[leosurwiki]

Gemfile cannot serve for transitive dependency version control, you might need to declare it in the gemspec https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/aws-sdk-core.gemspec

[mullermp]

That's true. We try not to add dependencies in core. If Oj is not installed, we fall back to JSON (https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/json.rb#L63) and we do similar things such as for XML (https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/xml/parser.rb#L68).

Edit - to clarify, we do it this way so that customers don't NEED oj or other XML parsing libraries, for example.

[leosurwiki]

Totally understandable, but it's kinda risky to have a dependency without version control unless it's very stable. Since this has been discovered as an issue, I wish it's possible to reevaluate this decision 🙏

[mullermp]

Unfortunately I don't think I can add it as a spec dependency anyway because it's a native C extension gem, and then JRuby customers would be broken. The implementation frees up what parser you may be using.

I'm thinking I can just add JSON::ParserError to the list here:

aws-sdk-ruby/gems/aws-sdk-core/lib/aws-sdk-core/json.rb

Line 53 in a99d0ab

[Oj::ParseError, EncodingError]

I'm open to other ideas.

[leosurwiki]

👍 that also sounds good to me

[mullermp]

Before I make this change, I'm trying to track down what version that was introduced in. I'm using the latest oj version 3.10.5 and trying to run Oj.load("<ServiceUnavailableException/>") but it raises an Oj::ParseError as expected..

[mullermp]

From that other issue, I tried using version 3.3.6 as well:

irb(main):001:0> require 'oj'
=> true
irb(main):002:0> Oj::VERSION
=> "3.3.6"
irb(main):003:0> Oj.load("<ServiceUnavailableException/>")
Traceback (most recent call last):
        5: from /Users/mamuller/.rbenv/versions/2.6.5/bin/irb:23:in `<main>'
        4: from /Users/mamuller/.rbenv/versions/2.6.5/bin/irb:23:in `load'
        3: from /Users/mamuller/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
        2: from (irb):3
        1: from (irb):3:in `load'
Oj::ParseError (unexpected character at line 1, column 1 [parse.c:690])

[leosurwiki]

This behavior might be related to another gem.
Could you also try again with:
oj_mimic_json (1.0.1)

[mullermp]

Recap from other thread - key component is mimic_JSON even when passed explicit options.

Aws> Oj.mimic_JSON
=> JSON
Aws> Oj.load("<ServiceUnavailableException/>", { mode: :compat, symbol_keys: false })
JSON::ParserError: Empty input () at line 1, column 1 [parse.c:995] in '<ServiceUnavailableException/>
from (pry):3:in `load'

[mullermp]

mimic_JSON will certainly complicate testing. It appears to rewrite the JSON namespace.