micropkg packaging improvements

[astrojuanlu]

Description

• Fix Stop relying on .egg-info directories for kedro micropkg #2567.
• Fix Document clearly thatkedro micropkg pull relies on sdists #2542.
• Beginning of Use pyproject.toml instead of setup.py for kedro micropkg #2414 (still generates a setup.py on the fly, but the code is ready to use a pyproject.toml instead).
• Obliterate any remaining usage of pkg_resources, which is deprecated and will go away any time.

Development notes

All the packaging stuff is quite complex - in fact, despite an initially good streak, this is now my second backtrack attempt, after opening gh-2569 and gh-2597 (hopefully those two can go next).

Keeping the micropackaging workflow intact while removing the strong assumptions around setup.py was extra hard, but I think I managed while staying within the "blessed" way of doing things. I could not avoid a runtime dependency on setuptools for now though, so we'll have to keep it around, maybe as an extra (gh-2350).

It's been mentioned that this workflow is weird and that we should evaluate whether we want to keep it or not https://github.com/kedro-org/kedro/milestone/21 but this work blocks the rest of the progress in https://github.com/kedro-org/kedro/milestone/36, so I didn't want to block it on that decision (which could be taken after 0.19).

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[astrojuanlu]

A small number of corner cases is failing, will have a look at those later this week.

[astrojuanlu]

The last set of test failures is due to a behavior change from pkg_resources to packaging, noted it here pypa/packaging#644 (comment)

I am hoping that we get an official answer soon, but while we wait, I think this is ready for review.

[stichbury]

Happy with this from docs perspective, but obvs not the main point of the PR.

[merelcht]

I'll need to look at this a second time, because there's quite a lot going on that I still need to process. I've left some initial thoughts/comments, but they're mostly nitpicks 🙈

General question is if this implementation is backwards compatible? (I'll also try it myself when re-reviewing)

[astrojuanlu]

I'll need to look at this a second time, because there's quite a lot going on that I still need to process. I've left some initial thoughts/comments, but they're mostly nitpicks 🙈

I totally understand... 2622da6 could be reviewed separately, it's what makes things more complicated here.

I could also split this in 2 PRs if it's easier for folks to review.

General question is if this implementation is backwards compatible? (I'll also try it myself when re-reviewing)

I didn't change any tests (only dropped one that was not needed anymore and modified error messages and pip invokations in others), so in principle this should be backwards compatible.

[astrojuanlu]

Test failures hopefully addressed by https://github.com/pypa/packaging/pull/696/files (although I don't want to wait on that for this PR, we'll probably have to hack around it)

[merelcht]

I've tried packaging and pulling with this new version and the old (current) one and it all seems to work fine and to be backwards compatible 👌 It seems like the micropackage that's generated is still exactly the same. I guess that's supposed to be the case right?

[astrojuanlu]

Yes, absolutely! That's why this PR would not fully close gh-2414 - the generated micropkg pretty much still uses setup.py.

[merelcht]

This looks like to be working as expected, so I'm happy to have it merged in. Maybe add to the release notes which changes were made here for traceability? And thanks for helping to improve this part of messy code base @astrojuanlu 🙇 ⭐

[astrojuanlu]

I have addressed all the outstanding comments and now I believe all tests should pass. If everything goes well and unless there are other opinions, I'll merge this.

[astrojuanlu]

All tests passing, CI failures due to non-100 % coverage. Will need to add tests to cover the extra possible (but unlikely) error messages.

[astrojuanlu]

Added tests, finally! This is ready to go in. It has 2 approvals already but waiting until EOD to merge.

[merelcht]

Added tests, finally! This is ready to go in. It has 2 approvals already but waiting until EOD to merge.

Ideally another engineer would approve this since the majority of the changes are in code.

[SajidAlamQB]

Overall, the changes you've made to the micro-packaging workflow seem comprehensive and well thought out. Just some minor comments about readability.

Thank you @astrojuanlu