Mishandled comments with extended mode regexp?

[owst]

My understanding (based on the docs) is that the following are equivalent:

  /
    a # bc+
  /x

  /
    a #bc+
  /x

  /
    a# bc+
  /x

  /
    a#bc+
  /x

  /a # bc+/x
  /a #bc+/x
  /a# bc+/x
  /a#bc+/x

in all cases, the pattern should match only a single 'a' as whitespace and anything on a line following a # is ignored. Indeed it seems that Ruby (2.6.6) does indeed treat them as equivalent. However, it seems that regexp_parser does not, and only recognises the comment if:

1. The pattern is across multiple lines (examples 1-4), and
2. There is a space before the # (examples 1-2)

To demonstrate, run the following:

require 'regexp_parser'

p Regexp::Parser::VERSION

[
  /
    a # bc+
  /x,
  /
    a #bc+
  /x,
  /
    a# bc+
  /x,
  /
    a#bc+
  /x,
  /a # bc+/x,
  /a #bc+/x,
  /a# bc+/x,
  /a#bc+/x
].each do |pat|
  puts "Pattern: #{pat.inspect}"
  pat =~ 'abcc'
  puts "pat =~ 'abccc' last_match: #{Regexp.last_match}"
  puts Regexp::Parser.parse(pat).each_expression.map { |exp, _| exp.class.name.sub(/.*::/, "") }.join(', ')

  puts
end

prints:

"1.7.1"
Pattern: /
    a # bc+
  /x
pat =~ 'abccc' last_match: a
WhiteSpace, Literal, WhiteSpace, Comment, WhiteSpace

Pattern: /
    a #bc+
  /x
pat =~ 'abccc' last_match: a
WhiteSpace, Literal, WhiteSpace, Comment, WhiteSpace

Pattern: /
    a# bc+
  /x
pat =~ 'abccc' last_match: a
WhiteSpace, Literal, WhiteSpace, Literal, Literal, WhiteSpace

Pattern: /
    a#bc+
  /x
pat =~ 'abccc' last_match: a
WhiteSpace, Literal, Literal, WhiteSpace

Pattern: /a # bc+/x
pat =~ 'abccc' last_match: a
Literal, WhiteSpace, Literal, Literal

Pattern: /a #bc+/x
pat =~ 'abccc' last_match: a
Literal, WhiteSpace, Literal, Literal

Pattern: /a# bc+/x
pat =~ 'abccc' last_match: a
Literal, WhiteSpace, Literal, Literal

Pattern: /a#bc+/x
pat =~ 'abccc' last_match: a
Literal, Literal

Notice how all patterns match just 'a', but only the first two parse include a Comment expression when parsed.

Am I misunderstanding something, or is this a bug?

[jaynetics]

@owst you were correct, this was a bug.

thank you for the excellent contribution!

i'll release a new version with your changes in the coming days.

[jaynetics]

released with v1.8.0