-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve Requirement/Marker parser with context-sensitive tokenisation #624
Commits on Dec 5, 2022
-
Convert
Token
into a dataclassThis makes it a fully-fleshed-out class for holding data.
Configuration menu - View commit details
-
Copy full SHA for 2ceccfc - Browse repository at this point
Copy the full SHA 2ceccfcView commit details -
Convert parser exception into a rich exception class
This also pulls out the error message formatting logic into the error itself.
Configuration menu - View commit details
-
Copy full SHA for a25e85f - Browse repository at this point
Copy the full SHA a25e85fView commit details -
Use a richer type for
Tokenizer.rules
This helps pyright better understand what's happening.
Configuration menu - View commit details
-
Copy full SHA for 09f31ff - Browse repository at this point
Copy the full SHA 09f31ffView commit details -
Provide dedicated
parse_{requirement,marker}(str)
functionsThese provide a consistent call signatures into the parser. This also decouples the tokenizer from the `Marker` class.
Configuration menu - View commit details
-
Copy full SHA for 1c930f1 - Browse repository at this point
Copy the full SHA 1c930f1View commit details -
Rename
req
toparsed
inRequirement.__init__
This makes it easier to read through the function, with a clearer name.
Configuration menu - View commit details
-
Copy full SHA for 650c7c6 - Browse repository at this point
Copy the full SHA 650c7c6View commit details -
Rename parser's
Requirement
toParsedRequirement
This draws a clear distinction between this and the user-visible `Requirement` object.
Configuration menu - View commit details
-
Copy full SHA for 282b4e1 - Browse repository at this point
Copy the full SHA 282b4e1View commit details -
Rework the parser with context-sensitive tokenisation
This reduces how many regex patterns would be matched against the input while also enabling the parser to resolve ambiguity in-place.
Configuration menu - View commit details
-
Copy full SHA for 07bf6f4 - Browse repository at this point
Copy the full SHA 07bf6f4View commit details -
Parse markers inline when parsing requirements
This allows for nicer error messages, which show the entire requirement string and highlight the marker in particular.
Configuration menu - View commit details
-
Copy full SHA for c6baf52 - Browse repository at this point
Copy the full SHA c6baf52View commit details -
Factor out parsing semicolon-marker for requirements
This eliminates a point of duplication and ensures that the error messaging is consistent.
Configuration menu - View commit details
-
Copy full SHA for 6b2f3de - Browse repository at this point
Copy the full SHA 6b2f3deView commit details -
Tweak the presentation of ParserSyntaxError spans
This makes it easier to identify what position the parser was checking, compared to relevant context to the reader.
Configuration menu - View commit details
-
Copy full SHA for 177e9ff - Browse repository at this point
Copy the full SHA 177e9ffView commit details -
Make URLs match "not whitespace"
This is more permitting and better handles tabs used as whitespace.
Configuration menu - View commit details
-
Copy full SHA for 1c3f900 - Browse repository at this point
Copy the full SHA 1c3f900View commit details -
Update
IDENTIFIER
to match PEP 508's stipulated syntaxThis follows what PEP 508's grammar says is a valid identifier.
Configuration menu - View commit details
-
Copy full SHA for 4a4d835 - Browse repository at this point
Copy the full SHA 4a4d835View commit details -
Make arbitrary version matching accept what
LegacySpecifier
didThis makes it possible for the arbitrary matches to be used within requirement specifiers without special constraints.
Configuration menu - View commit details
-
Copy full SHA for 39ae524 - Browse repository at this point
Copy the full SHA 39ae524View commit details -
Better reflect what is optional within
specifier
/version_many
This makes it clearer in the docstring grammars that a name without any specifier is valid.
Configuration menu - View commit details
-
Copy full SHA for 97e7649 - Browse repository at this point
Copy the full SHA 97e7649View commit details -
Flatten nested ifs into if-elif
This makes the control flow slightly easier to understand.
Configuration menu - View commit details
-
Copy full SHA for 92b9545 - Browse repository at this point
Copy the full SHA 92b9545View commit details -
Rewrite test suite for requirements parsing
This now exercises more edge cases and validates that the error messages are well-formed.
Configuration menu - View commit details
-
Copy full SHA for 3a7cdb6 - Browse repository at this point
Copy the full SHA 3a7cdb6View commit details
Commits on Dec 6, 2022
-
Improve error message for bad version specifiers in
Requirement
This makes it easier to understand what the state of the parser is and what is expected at that point.
Configuration menu - View commit details
-
Copy full SHA for 0399eaf - Browse repository at this point
Copy the full SHA 0399eafView commit details -
Add
ParserSyntaxError
as the cause ofInvalid{Requirement/Marker}
This ensures that these error tracebacks correctly describe the causality between the two errors.
Configuration menu - View commit details
-
Copy full SHA for 83aae66 - Browse repository at this point
Copy the full SHA 83aae66View commit details
Commits on Dec 7, 2022
-
Permit whitespace around
marker_atom
This ensures that a marker with whitespace around it is parsed correctly.
Configuration menu - View commit details
-
Copy full SHA for 163993a - Browse repository at this point
Copy the full SHA 163993aView commit details -
This is better aligned with the naming from PEP 508.
Configuration menu - View commit details
-
Copy full SHA for fa4b69d - Browse repository at this point
Copy the full SHA fa4b69dView commit details -
Enforce word boundaries in operators and names
This ensures that these are only parsed when they're independent words.
Configuration menu - View commit details
-
Copy full SHA for ff75da7 - Browse repository at this point
Copy the full SHA ff75da7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4945856 - Browse repository at this point
Copy the full SHA 4945856View commit details -
Permit arbitrary whitespace around versions specifier in parenthesis
This is more consistent with the rest of the format which is largely whitespace agnostic.
Configuration menu - View commit details
-
Copy full SHA for 7869a1a - Browse repository at this point
Copy the full SHA 7869a1aView commit details