Handwritten parser for parsing requirements #484

hrnciar · 2021-11-03T11:52:43Z

Hello,

I wrote a parser for packaging that should allow us to remove the dependency on pyparsing.
I am opening this as a draft as this is my first PR into packaging and I would love to hear feedback from you on what to improve in my PR.

Thank you.

Fixes: #468

brettcannon · 2021-11-03T19:07:02Z

Two quick things (I haven't really reviewed any of the code).

One, looks like there are some CI failures.

Two, do we want packaging.parser and packaging.tokenizer to be public? They strike me as implementation details, so we would probably prefer to keep them as private (i.e. leading underscore in the module names).

pradyunsg · 2021-11-03T19:09:48Z

+1 to both the things that Brett said.

pradyunsg · 2021-11-03T19:12:09Z

That CI failures seem to be because of the use of := (sorry, you can't have nice things like that here) and because a few branches of the parser aren't covered in tests.

packaging/parser.py

packaging/requirements.py

pradyunsg · 2021-11-03T19:16:19Z

All that said, this is super cool and I'm very excited to see this progressing! Thanks for picking this up @hrnciar! ^>^

packaging/_parser.py

pradyunsg · 2021-11-13T09:25:13Z

Pulling this out of draft status, given there is a review request.

pradyunsg

This is pretty much a desk review -- I haven't cloned and poked at this yet; I'll do that once the linters are happy with this PR.

I don't think I have anything to say about the parsing logic other than that it looks correct, and that CI being happy is a good sign. :)

I think we can/should bundle this change with the legacy version removal, which... I'll pick back up once this lands.

packaging/_parser.py

packaging/_tokenizer.py

packaging/_parser.py

pradyunsg · 2021-11-18T07:03:13Z

A gentle ping.

encukou · 2021-11-18T15:37:52Z

packaging/_parser.py

+MarkerType = Tuple[Union[Variable, Value], Op, Union[Variable, Value]]
+# MarkerAtom = Union[MarkerType, List["MarkerAtom"]]


These names are a bit confusing. Perhaps use MarkerAtom, MarkerItem, MarkerVar and MarkerList, corresponding with their parse rules?

I realized that I forget to rename the names just after I pushed it. I took into account your latest comment and pushed a new change, PTAL.

packaging/_parser.py

pradyunsg · 2021-11-18T16:02:38Z

(FYI: The CI isn't running due to merge conflicts with main)

pradyunsg · 2022-07-08T14:28:17Z

packaging/_tokenizer.py

+    def raise_syntax_error(self, *, message: str) -> NoReturn:
+        """Raise SyntaxError at the given position in the marker"""
+        at = f"at position {self.position}:"
+        marker = " " * self.position + "^"
+        raise ParseExceptionError(
+            f"{message}\n{at}\n    {self.source}\n    {marker}",
+            self.position,
+        )


I really like this! :)

brettcannon

One of the few things that Black can't check syntactically, the docstrings unfortunately don't match the style in the rest of the project (both the newline structure and the sentences need to end with proper punctuation).

hrnciar · 2022-07-11T15:31:17Z

I've addressed some remarks, but there is still some stuff to do. I am heading to EuroPython tomorrow so I will continue next week when I am back. If anyone would like to meet in Dublin and have a chat IRL feel free to ping me :).

encukou · 2022-07-11T16:09:32Z

packaging/_parser.py

+
+def parse_named_requirement(requirement: str) -> Requirement:
+    """
+    named_requirement: IDENTIFIER extras (URL_SPEC | specifier) (END | SEMICOLON marker_expr END)?


Suggested change

named_requirement: IDENTIFIER extras (URL_SPEC | specifier) (END | SEMICOLON marker_expr END)?

named_requirement: IDENTIFIER extras (URL_SPEC | specifier) (SEMICOLON marker_expr )? END

pradyunsg

LGTM, assuming @encukou's suggestion is applied!

And... you pacify flake8 -- in whatever manner you deem appropriate.

Tokenizer uses _version_regex_str to detect 'VERSION' token.

hrnciar · 2022-07-19T11:39:52Z

I've improved the docstrings and fixed a couple of nitpicks. Unless there is something else to improve, I think, we are done.

hrnciar · 2022-07-28T09:19:15Z

@brettcannon could you please do a review? Is there something specific I should improve? I'd like to finish this PR. Thank you.

brettcannon · 2022-07-29T18:01:02Z

@hrnciar it's in my review queue, but if @pradyunsg wants to merge w/o a review for me then I am totally fine with that.

hroncok · 2022-08-04T12:47:32Z

Changelog entry: #581

ofek · 2022-09-03T15:29:41Z

This is great for CLIs too 🙂 pypa/hatch#449 (comment)

brettcannon added enhancement packaging.version packaging.markers packaging.requirements packaging.specifiers and removed packaging.version labels Nov 3, 2021

pradyunsg reviewed Nov 3, 2021

View reviewed changes

packaging/parser.py Outdated Show resolved Hide resolved

pradyunsg reviewed Nov 3, 2021

View reviewed changes

packaging/requirements.py Outdated Show resolved Hide resolved

packaging/requirements.py Outdated Show resolved Hide resolved

hrnciar force-pushed the parser branch 3 times, most recently from fe754f3 to 3062470 Compare November 5, 2021 15:22

uranusjr reviewed Nov 7, 2021

View reviewed changes

packaging/_parser.py Outdated Show resolved Hide resolved

hrnciar force-pushed the parser branch from 3062470 to c468a1a Compare November 8, 2021 08:52

hrnciar requested a review from pradyunsg November 9, 2021 07:20

hroncok reviewed Nov 11, 2021

View reviewed changes

packaging/_parser.py Outdated Show resolved Hide resolved

hrnciar force-pushed the parser branch from c468a1a to 721ca94 Compare November 11, 2021 14:33

pradyunsg marked this pull request as ready for review November 13, 2021 09:25

pradyunsg reviewed Nov 13, 2021

View reviewed changes

hrnciar force-pushed the parser branch from 721ca94 to c242570 Compare November 18, 2021 15:30

encukou reviewed Nov 18, 2021

View reviewed changes

uranusjr reviewed Nov 18, 2021

View reviewed changes

packaging/_parser.py Outdated Show resolved Hide resolved

hrnciar force-pushed the parser branch from c242570 to f98902d Compare November 18, 2021 16:00

hrnciar force-pushed the parser branch from f98902d to 7919b5e Compare November 18, 2021 16:09

pradyunsg reviewed Jul 8, 2022

View reviewed changes

pradyunsg mentioned this pull request Jul 8, 2022

Release 22.0 #569

Closed

brettcannon requested changes Jul 8, 2022

View reviewed changes

hrnciar force-pushed the parser branch from 36be7b6 to 9bb4761 Compare July 11, 2022 07:36

Remove old pyparsing code

5f44d42

hrnciar force-pushed the parser branch 3 times, most recently from 5a961ae to 4f52bac Compare July 11, 2022 15:25

encukou reviewed Jul 11, 2022

View reviewed changes

pradyunsg requested review from pradyunsg and brettcannon July 11, 2022 22:47

pradyunsg approved these changes Jul 11, 2022

View reviewed changes

hrnciar force-pushed the parser branch 3 times, most recently from 6c8ed42 to a29816f Compare July 19, 2022 10:38

hrnciar added 4 commits July 19, 2022 13:32

Add sources for new parser and tokenizer

b1861b0

Split Specifier regex into operator and version parts

6ceecf6

Tokenizer uses _version_regex_str to detect 'VERSION' token.

Changes to integrate new parser into packaging + test adjustments

32a3f32

Normalize extra values in markers

c30f323

hrnciar force-pushed the parser branch from a29816f to c30f323 Compare July 19, 2022 11:32

Merge branch 'main' into parser

01e0645

pradyunsg merged commit 2e5593c into pypa:main Jul 31, 2022

pradyunsg mentioned this pull request Dec 6, 2022

Improve Requirement/Marker parser with context-sensitive tokenisation #624

Merged

j01101111sh mentioned this pull request Dec 8, 2022

Bump packaging from 21.3 to 22.0 j01101111sh/metabase-tools#178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handwritten parser for parsing requirements #484

Handwritten parser for parsing requirements #484

hrnciar commented Nov 3, 2021

brettcannon commented Nov 3, 2021

pradyunsg commented Nov 3, 2021

pradyunsg commented Nov 3, 2021

pradyunsg commented Nov 3, 2021

pradyunsg commented Nov 13, 2021

pradyunsg left a comment

pradyunsg commented Nov 18, 2021

encukou Nov 18, 2021

hrnciar Nov 18, 2021

pradyunsg commented Nov 18, 2021

pradyunsg Jul 8, 2022

brettcannon left a comment

hrnciar commented Jul 11, 2022

encukou Jul 11, 2022

pradyunsg left a comment •

edited

hrnciar commented Jul 19, 2022

hrnciar commented Jul 28, 2022

brettcannon commented Jul 29, 2022

hroncok commented Aug 4, 2022

ofek commented Sep 3, 2022

		MarkerType = Tuple[Union[Variable, Value], Op, Union[Variable, Value]]
		# MarkerAtom = Union[MarkerType, List["MarkerAtom"]]

	named_requirement: IDENTIFIER extras (URL_SPEC \| specifier) (END \| SEMICOLON marker_expr END)?
	named_requirement: IDENTIFIER extras (URL_SPEC \| specifier) (SEMICOLON marker_expr )? END

Handwritten parser for parsing requirements #484

Handwritten parser for parsing requirements #484

Conversation

hrnciar commented Nov 3, 2021

brettcannon commented Nov 3, 2021

pradyunsg commented Nov 3, 2021

pradyunsg commented Nov 3, 2021

pradyunsg commented Nov 3, 2021

pradyunsg commented Nov 13, 2021

pradyunsg left a comment

Choose a reason for hiding this comment

pradyunsg commented Nov 18, 2021

encukou Nov 18, 2021

Choose a reason for hiding this comment

hrnciar Nov 18, 2021

Choose a reason for hiding this comment

pradyunsg commented Nov 18, 2021

pradyunsg Jul 8, 2022

Choose a reason for hiding this comment

brettcannon left a comment

Choose a reason for hiding this comment

hrnciar commented Jul 11, 2022

encukou Jul 11, 2022

Choose a reason for hiding this comment

pradyunsg left a comment • edited

Choose a reason for hiding this comment

hrnciar commented Jul 19, 2022

hrnciar commented Jul 28, 2022

brettcannon commented Jul 29, 2022

hroncok commented Aug 4, 2022

ofek commented Sep 3, 2022

pradyunsg left a comment •

edited