Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding require to cli breaks rouge #3623

Closed
scphantm opened this issue Apr 16, 2020 · 16 comments
Closed

adding require to cli breaks rouge #3623

scphantm opened this issue Apr 16, 2020 · 16 comments
Assignees
Labels

Comments

@scphantm
Copy link

I have a doc that runs perfectly fine with the latest of everything and rouge. ruby 2.6.0, adoc 2.0.10, rogue 3.18.0,

now everything runs fine until i attempt to add a new extension with the --require= switch on the cli. When i do that, i get this

/usr/lib/ruby/gems/2.6.0/gems/rouge-3.18.0/lib/rouge.rb:54:in `block in load_lexers': asciidoctor: FAILED: /documents/index.adoc: Failed to load AsciiDoc document - uninitialized constant #<Class:Rouge>::Lexers (NameError)
        from /usr/lib/ruby/gems/2.6.0/gems/rouge-3.18.0/lib/rouge.rb:53:in `each'
        from /usr/lib/ruby/gems/2.6.0/gems/rouge-3.18.0/lib/rouge.rb:53:in `load_lexers'
        from /usr/lib/ruby/gems/2.6.0/gems/rouge-3.18.0/lib/rouge.rb:69:in `<top (required)>'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `require'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:130:in `rescue in require'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:34:in `require'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/rouge_ext.rb:2:in `<top (required)>'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/helpers.rb:27:in `require_library'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/syntax_highlighter/rouge.rb:78:in `load_library'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/syntax_highlighter/rouge.rb:74:in `library_available?'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/syntax_highlighter/rouge.rb:12:in `highlight?'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/substitutors.rb:1288:in `commit_subs'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:918:in `next_block'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:379:in `next_section'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:364:in `next_section'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:364:in `next_section'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:97:in `parse'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/document.rb:549:in `parse'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/load.rb:83:in `load'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:78:in `convert'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:183:in `block in convert_file'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:183:in `open'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:183:in `convert_file'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/cli/invoker.rb:128:in `block in invoke!'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/cli/invoker.rb:111:in `each'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/cli/invoker.rb:111:in `invoke!'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/bin/asciidoctor:15:in `<top (required)>'
        from /usr/bin/asciidoctor:23:in `load'
        from /usr/bin/asciidoctor:23:in `<main>'
/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- rouge (LoadError)
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/rouge_ext.rb:2:in `<top (required)>'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/helpers.rb:27:in `require_library'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/syntax_highlighter/rouge.rb:78:in `load_library'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/syntax_highlighter/rouge.rb:74:in `library_available?'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/syntax_highlighter/rouge.rb:12:in `highlight?'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/substitutors.rb:1288:in `commit_subs'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:918:in `next_block'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:379:in `next_section'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:364:in `next_section'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:364:in `next_section'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/parser.rb:97:in `parse'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/document.rb:549:in `parse'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/load.rb:83:in `load'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:78:in `convert'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:183:in `block in convert_file'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:183:in `open'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/convert.rb:183:in `convert_file'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/cli/invoker.rb:128:in `block in invoke!'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/cli/invoker.rb:111:in `each'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/lib/asciidoctor/cli/invoker.rb:111:in `invoke!'
        from /usr/lib/ruby/gems/2.6.0/gems/asciidoctor-2.0.10/bin/asciidoctor:15:in `<top (required)>'
        from /usr/bin/asciidoctor:23:in `load'
        from /usr/bin/asciidoctor:23:in `<main>'

the extension im trying to add is the tree-block-macro.rb from the asciidoc extensions lab. If i run the sample file in the extension alone, it runs fine. but the moment i add the :source-highlighter: rouge to the sample file, it blows up. I'm not a ruby expert, but it looks to me like instead of --require= adding to the load path, its overwriting the load path and other gems are being loaded wrong.

@mojavelinux
Copy link
Member

I'll investigate.

its overwriting the load path and other gems are being loaded wrong.

Nope, it just does this:

require path

But is possible that Rouge itself is doing what you're suggesting.

@mojavelinux
Copy link
Member

You can work around the problem by forcing Rouge to load:

asciidoctor -r rouge -a source-highlighter=rouge -r path/to/tree-block-macro.rb doc.adoc

@mojavelinux
Copy link
Member

Ultimately, this is a bug in Rouge. It's incorrect to require libraries using load in place of require. It causes programs that rely on the behavior of require to malfunction. Please report this upstream. The workaround in Asciidoctor is to force Rouge to load by passing -r rouge.

@danyill
Copy link

danyill commented Apr 16, 2020 via email

@mojavelinux
Copy link
Member

Here's a shim I came up with:

unless defined? Rouge.version
  begin
    require 'rouge'
  rescue
    old_verbose, $VERBOSE = $VERBOSE, nil
    original_load = method :load
    def load path
      require path
    end
    require 'rouge'
    define_method :load, original_load
    $VERBOSE = old_verbose
  end
end

I don't really see any other way around this since what Rouge is doing is fundamentally an incorrect use of load. It's also leaking a development concern (live reload for lexer development) into production code.

cc: @pyrmont

@mojavelinux
Copy link
Member

mojavelinux commented Apr 16, 2020

I know why this is happening with extensions specifically. It's caused by the line include Asciidoctor and the fact that Asciidoctor defines a method method named load (which then conflicts with the main load method). (It wouldn't be a problem if Rouge called Kernel.load instead of load).

So this is partially Asciidoctor's fault in that it should not be defining a module method named load that can interfere with the main load method. I'll take that up as a separate action item.

@mojavelinux
Copy link
Member

Here's a simpler solution:

unless defined? Rouge.version
  define_method :load, &(Kernel.method :load)
  require 'rouge'
end

@jneen
Copy link

jneen commented Apr 17, 2020

I suggest you open a ticket on Rouge instead of pinging individual Rouge developers. load is standard, and we do in fact use it in a safe manner. We're happy to take a look at it for users who require load to be something else at the global top-level.

@danyill
Copy link

danyill commented Apr 17, 2020

There was this upstream comment when the issue was previously raised: rouge-ruby/rouge#886 (comment)

@jneen
Copy link

jneen commented Apr 17, 2020

That is a separate issue. Using load internally to Rouge doesn't effect anyone outside unless they override the global load method, which is generally not a great idea. As that comment suggests, we use load internally to enable quick lexer development, and in the near future it will also significantly reduce the memory footprint of our library through lazy loading. I'm happy to fix this by using Kernel.load in its place, but I also ask for the developers of this project to reconsider having a load method in the global namespace, as it is likely to break on other libraries as well.

@jneen
Copy link

jneen commented Apr 17, 2020

In the future, @mojavelinux, please refrain from pinging individual developers. If you open an issue on our tracker, we will happily triage it with the rest of the issues. Our project is volunteer-based and we can't prioritize your needs over others on a whim.

@jneen
Copy link

jneen commented Apr 17, 2020

The bug appears to be here, where every single method in the Asciidoctor namespace is leaked into the global scope:

https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/tree-block-macro/extension.rb#L4

It should be sufficient to move that line into the class context that comprises the rest of the file.

@mojavelinux
Copy link
Member

mojavelinux commented Apr 19, 2020

I also ask for the developers of this project to reconsider having a load method in the global namespace, as it is likely to break on other libraries as well.

I acknowledge that the side effect is being introduced from the Asciidoctor side and I'm taking that up as an action item to mitigate as well.

The bug appears to be here, where every single method in the Asciidoctor namespace is leaked into the global scope:

That statement was only meant to import the classes into the global scope, but the introduction of module_function in Asciidoctor 2 causes it to also bring in methods. Hence, we end up with this rather unpleasant side effect.

It's unfortunate that the global load function can be so easily overridden without a warning. Otherwise, we would have seen this potential for conflict when it was introduced.

^ In fact, it's very inconsistent that Ruby doesn't produce a warning when warnings are enabled (same is true for both load and require). I wonder why Ruby decided this was an exception. I guess that's just something we need to be aware of.

@mojavelinux
Copy link
Member

In the future, please refrain from pinging individual developers. If you open an issue on our tracker, we will happily triage it with the rest of the issues. Our project is volunteer-based and we can't prioritize your needs over others on a whim.

@jneen I had an ongoing discussion with Michael about this issue and I was pinging him by request to let him know if this happened to come up again...and it did. It's rather odd you're replying to a ping that wasn't even directed at you.

Our project is also volunteer-based and I know very well that needs of individual community members cannot be prioritized on a whim. Nothing I did suggested I was asking for that treatment. So I kindly ask you to refrain from coming in this project and making accusations. Your comment was uncalled for.

@mojavelinux
Copy link
Member

A change has been proposed in Rouge to use Kernel.load instead of load since, as we've learned, the load method can get replaced without any warning. See rouge-ruby/rouge#1503

I've opened a separate issue in Asciidoctor to drop the use of module_function. See #3625.

The short term fix for this issue is to avoid using include Asciidoctor as it causes the load method in Ruby to be replaced.

@mojavelinux mojavelinux self-assigned this Apr 19, 2020
@pyrmont
Copy link

pyrmont commented Apr 20, 2020

@scphantm I don't know if you were following the discussion in the Rouge PR but we've updated our calls to load to be qualified calls to Kernel::load. That update will go out in the upcoming v3.19.0 gem. On our monthly cadence, that's scheduled for release on Tuesday 12 May. If you need the fix straightaway, you can pull the gem in from Rouge's current master branch :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants