Skip to content

Commit

Permalink
- dedenter.rb: Fix squiggly heredoc line continuation handling (#819)
Browse files Browse the repository at this point in the history
Firstly, this fixes an issue with non-interpolating squiggly heredocs, where line continuation characters should be ignored. Two things were broken:

- The line continuation backslash would be removed, as well as the subsequent carriage return
- The subsequent line would not be dedented.

For example, the following source:

```ruby
<<~'END'
  foo \
  bar
END
```

Would be parsed as follows:

```
(dstr
  (str "foo ")
  (str "  bar\n"))
```

With the change, the result is:

```
(dstr
  (str "foo \\\n")
  (str "bar\n"))
```

This also fixes an issue with interpolating squiggly heredocs, where an escaped backslash followed by a newline would be interpreted as a line continuation. This is issue #789.

In that case, the following source:

```ruby
<<~END
  foo \\
  bar
END
```

Would (again) be parsed as follows:

```
(dstr
  (str "foo ")
  (str "  bar\n"))
```

With the change, the result is:

```
(dstr
  (str "foo \\\n")
  (str "bar\n"))
```
  • Loading branch information
mvz committed Sep 24, 2021
1 parent 22e6ca5 commit 61bf873
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 1 deletion.
8 changes: 7 additions & 1 deletion lib/parser/lexer/dedenter.rb
Expand Up @@ -38,7 +38,13 @@ def dedent(string)
# Prevent the following error when processing binary encoded source.
# "\xC0".split # => ArgumentError (invalid byte sequence in UTF-8)
lines = string.force_encoding(Encoding::BINARY).split("\\\n")
lines.map! {|s| s.force_encoding(original_encoding) }
if lines.length == 1
# If the line continuation sequence was found but there is no second
# line, it was not really a line continuation and must be ignored.
lines = [string]
else
lines.map! {|s| s.force_encoding(original_encoding) }
end

if @at_line_begin
lines_to_dedent = lines
Expand Down
16 changes: 16 additions & 0 deletions test/test_parser.rb
Expand Up @@ -439,6 +439,22 @@ def test_parser_bug_640
SINCE_2_3)
end

def test_dedenting_non_interpolating_heredoc_line_continuation
assert_parses(
s(:dstr, s(:str, "baz\\\n"), s(:str, "qux\n")),
%Q{<<~'FOO'\n baz\\\n qux\nFOO},
%q{},
SINCE_2_3)
end

def test_dedenting_interpolating_heredoc_fake_line_continuation
assert_parses(
s(:dstr, s(:str, "baz\\\n"), s(:str, "qux\n")),
%Q{<<~'FOO'\n baz\\\\\n qux\nFOO},
%q{},
SINCE_2_3)
end

# Symbols

def test_symbol_plain
Expand Down

0 comments on commit 61bf873

Please sign in to comment.