Load micropackage dependencies statically

[astrojuanlu]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

This avoids building the wheel in an isolated environment just to extract the dependencies.

Cons: It relies on a PEP-621 compliant pyproject.toml file to be present, which only happens for Kedro projects created after #2280 was closed. Therefore, it is backwards incompatible.

Pros: It makes kedro micropkg pull fast again, cutting the time to run the full test suite in ~half (from 12 to 7 minutes on Linux, from 14 to 6 minutes on Windows 🎉)

Development notes

This uses pyproject-metadata instead of PyPA/build, see https://pypi.org/project/pyproject-metadata/

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[noklam]

Is it possible to keep both, using the Metadata when available and fallback if not?

[astrojuanlu]

Yes indeed!

[astrojuanlu]

I don't have time to pursue the backwards-compatible approach, leaving this as a POC in case anybody wants to pick it up. Essentially this would need

• To retain build as a dependency,
• Conditionally try to locate the PEP-621 metadata in pyproject.toml and otherwise use the slow codepath, and
• Test both things.

[merelcht]

@astrojuanlu Would it make sense to write this up in a ticket instead of keeping a PR open that will become more and more outdated? It's not super likely that this will be merged as is since it's from August and solving conflicts might be a real pain.

[astrojuanlu]

Good idea @merelcht , will do!

[astrojuanlu]

Closing in favour of #3457