New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for all valid string literals #115
Conversation
Pull Request Test Coverage Report for Build 181
💛 - Coveralls |
1 similar comment
Pull Request Test Coverage Report for Build 181
💛 - Coveralls |
Pull Request Test Coverage Report for Build 191
💛 - Coveralls |
blib2to3/pgen2/tokenize.py
Outdated
|
||
_strprefixes = set(combinations('r', 'R', 'f', 'F') + | ||
combinations('r', 'R', 'b', 'B') + | ||
['u', 'U']) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're missing 'uR', 'UR', 'Ur', 'ur'
which is valid on Python 2.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uR right 🤣 I was wondering why tokenize used to accept this in the regex above.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Funnily enough, since we are still supporting u""
in Python 3, not supporting ur""
is likely just an omission of whoever brought u""
back :-)
blib2to3/pgen2/tokenize.py
Outdated
return [(x + y).strip() for x in l for y in l + (" ",) if x.casefold() != y.casefold()] | ||
def _combinations(*l): | ||
return set( | ||
x + y for x in l for y in l + ("",) if x.casefold() != y.casefold() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That needs to be a (" ",)
in the middle
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh nevermind
Thanks! ✨ 🍰 ✨ |
Fixes #3. This was the cause for black not being able to support
rf
strings. Now this forked version oflib2to3
only parses valid string literals as defined by https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals