Fix string parsing with newline #123
Conversation
When looping over a joined str, if a node is ast.Str and the value is
just a newline "\n", the write() function adds an additional indentation
after it, which fails to represent the original string. By calling
self.result.append() here directly the issue is resolved.
The added test could show the issue. With code_gen unmodifed, it fails
with the following error:
```
AssertionError: "if 1:\n x = f'{host}\\n\\t{port}\\n '" != "if
1:\n x = f'{host}\\n\\t{port}\\n'"
if 1:
- x = f'{host}\n\t{port}\n '?
----
+ x = f'{host}\n\t{port}\n'
```
Which is exactly the problem.
This fixes parsing issues with many of Python 3.7's stdlib. Namely the
following ones:
/usr/lib/python3.7/warnings.py
/usr/lib/python3.7/netrc.py
/usr/lib/python3.7/test/test_embed.py
/usr/lib/python3.7/test/support/testresult.py
/usr/lib/python3.7/idlelib/grep.py
astor/code_gen.py
Outdated
| @@ -566,7 +566,7 @@ def visit_Str(self, node, is_joined=False): | |||
| def recurse(node): | |||
| for value in node.values: | |||
| if isinstance(value, ast.Str): | |||
| self.write(value.s) | |||
| self.result.append(value.s) | |||
There was a problem hiding this comment.
'\n' is already special-cased in the write() method:
Lines 171 to 172 in d9f6679
What do you think about fixing this in the write() method? (or even in the newline() method) Perhaps we could handle the '\n' case in the newline() method:
if node is None and extra == 0:
self.result.append('\n')Perhaps we could even add a new without_indentation keyword argument to minimize breaking user code?
There was a problem hiding this comment.
I'm a bit unsure why it was special-cased before - If I remove the special case at all, all the tests pass including the new one. I tried to look up git history but didn't find valuable info :(
There was a problem hiding this comment.
Got it now - so I think writing self.newline in the cases where indentation is needed would be more explicit. Tests green again :)
It should only consider spaces in the left
There was a problem hiding this comment.
I think this looks pretty good. Could you please fix the merge conflict? (sorry about that, by the way! I just merged #125.)
|
Fixed :) |
|
Thank you! |
When looping over a joined str, if a node is ast.Str and the value is
just a newline "\n", the write() function adds an additional indentation
after it, which fails to represent the original string. By calling
self.result.append() here directly the issue is resolved.
The added test could show the issue. With code_gen unmodifed, it fails
with the following error:
Which is exactly the problem.
This fixes parsing issues with many of Python 3.7's stdlib. Namely the
following ones:
/usr/lib/python3.7/warnings.py
/usr/lib/python3.7/netrc.py
/usr/lib/python3.7/test/test_embed.py
/usr/lib/python3.7/test/support/testresult.py
/usr/lib/python3.7/idlelib/grep.py