New Python handler is here

[pawamoy]

Hello everyone!

I've been working on a new Python handler. This issue serves as a public TODO list for what is needed before getting it out of the "experimental" state.

The new handler is mkdocstrings/python, and the old one has been extracted into mkdocstrings/python-legacy. The new data collector, mkdocstrings/griffe, is an improved version of mkdocstrings/pytkdocs. It's able to both visit (AST) and inspect (exec + introspection) objects, and has an overall better design.

Phase 1

• Finish implementing support for namespace packages in griffe
• Publish a new griffe release
• Fix docstrings admonitions rendering in mkdocstrings/python
• Use the new mkdocstrings_handlers namespace in mkdocstrings/python
• Publish a new mkdocstrings/python release
• Open mkdocstrings PR to allow it to load handlers from the mkdocstrings_handlers namespace (with backward-compatibility and deprecation warnings) refactor: Extract the Python handler into its own repository #356 and Support new mkdocstrings_handlers namespace #367
• Update PR refactor: Extract the Python handler into its own repository #356 with the python experimental extra actually installing mkdocstrings-python
• Update docs about the new handler: things that can possibly stop working (dynamic objects, custom templates?), deprecations, user warnings, how to ignore them, etc. docs: Document migration to the new Python handler #371
• Add user warning (specify extra) when using mkdocstrings/python-legacy
• Publish new legacy handler release
• Merge all mkdocstrings PRs
• Publish a new mkdocstrings release
• At this point users can start testing the new handler by depending on mkdocstrings[python]
• Add back usage docs on the legacy handler docs
• Enjoy a round of bugtracker cleaning 🤤

---

Phase 2

• Stop depending on the legacy handler in mkdocstrings
• Remove user warning when using mkdocstrings/python-legacy (since by now the extra is mandatory)
• Use the new mkdocstrings_handlers namespace in mkdocstrings/python-legacy as well
• Refactor mkdocstrings to get rid of the BaseCollector and BaseRenderer classes: merge everything into the BaseHandler class refactor: Deprecate BaseCollector and BaseRenderer #413
• Stop splitting options into selection and rendering keys. Keep backward compatibility (with deprecation warnings). refactor: Deprecate 'selection' and 'rendering' YAML keys #420
• Approximate feature parity between old and new handler (selection and rendering options)
• Document all the changes: docs: Update docs #429
• Publish a new release of mkdocstrings
• Publish a new release of mkdocstrings-python (it must depend on the new mkdocstrings version)
• Publish a new release of mkdocstrings-python-legacy (it must depend on the new mkdocstrings version)
• Enjoy a second round of bugtracker cleaning 🤤

---

Phase 3

• Refine griffe's ability to selectively inspect some objects (based on user configuration). Doable with extensions
• Bring some third-party libraries support in griffe (django/pydantic/marshmallow/attrs extensions). Extension system working (still has some limitations but it's a good start), we have a prototype for Pydantic.
• Use jinja blocks everywhere in mkdocstrings/python templates. Blocks were added, see https://mkdocstrings.github.io/python/usage/customization/#available-blocks

---

Phase 4

• Add a deprecation warning in mkdocstrings/python-legacy
• Progressively remove deprecated stuff. It's been more than a year, deprecated parts in mkdocstrings are removed now refactor: Remove deprecated parts #590

---

You are free to comment on the list items in this issue 🙂

[FasterSpeeding]

Since phase two has "Approximate feature parity between old and new handler" ticked as complete is there any rough idea of when the missed feature inherited_members might be implemented (even just phase wise)?

[pawamoy]

Good question! Well it will be implemented before the legacy handler is marked as deprecated, so during the current phase (3). Not sure exactly when though 🙂 Lots of work to do these days!

[pawamoy]

We're close to the end of the transition period.
I'll not mark yet the legacy handler as deprecated, since some projects are still using it, and have issues with the new handler. I want to fix these issues in the new handler before deprecating the legacy one.