Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Pytkdocs Loader assumes '_meta' attribute of classes has a 'get_fields' method #133

Closed
jaspergreener opened this issue Feb 3, 2022 · 5 comments
Closed

[BUG] Pytkdocs Loader assumes '_meta' attribute of classes has a 'get_fields' method #133

jaspergreener opened this issue Feb 3, 2022 · 5 comments

Comments

@jaspergreener
Copy link

Describe the bug
I am trying to use mkdocs + mkdocstrings to generate documentation for my codebase. My codebase uses graphene for setting up a GraphQL backend.

mkdocstrings cannot generate documentation based on docstring for classes that inherit from any of graphene's types, due to AttributeErrors:

(venv3.8) user@user-pc:~/PycharmProjects/my-project$ mkdocs serve
INFO     -  Building documentation...
INFO     -  Cleaning site directory
ERROR    -  mkdocstrings.extension: 'ObjectTypeOptions' object has no attribute 'get_fields'
            Traceback (most recent call last):
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/cli.py", line 205, in main
                output = json.dumps(process_json(line))
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/cli.py", line 114, in process_json
                return process_config(json.loads(json_input))
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/cli.py", line 91, in process_config
                obj = loader.get_object_documentation(path, members)
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/loader.py", line 358, in get_object_documentation
                root_object = self.get_module_documentation(leaf, members)
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/loader.py", line 436, in get_module_documentation
                root_object.add_child(self.get_module_documentation(leaf))
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/loader.py", line 436, in get_module_documentation
                root_object.add_child(self.get_module_documentation(leaf))
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/loader.py", line 426, in get_module_documentation
                root_object.add_child(self.get_class_documentation(child_node))
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/loader.py", line 542, in get_class_documentation
                if self.detect_field_model(attr_name, direct_members, all_members):
              File "/home/user/PycharmProjects/my-project/venv3.8/lib/python3.8/site-packages/pytkdocs/loader.py", line 577, in detect_field_model
                if remainder and not attrgetter(remainder)(all_members[first_order_attr_name]):
            AttributeError: 'ObjectTypeOptions' object has no attribute 'get_fields'
ERROR    -  Error reading page 'full_reference.md':
ERROR    -  Could not collect 'github_bug'

The attribute error is thrown because pytkdocs' Loader tries to interpret a '_meta' attribute as a Django meta field, see pytkdocs/loader.py:539: ("_meta.get_fields", ["django-model"], self.get_django_field_documentation).

In my case, there is a '_meta' field, but it has no 'get_fields' attribute.

To Reproduce
github_bug.py:

import graphene

class ExampleClass(graphene.ObjectType):
    """ You will never document me!!! """
    pass

index.md:

# Reference

::: github_bug

Generate documentation with mkdoc using mkdoc build or mkdoc serve

Expected behavior
I can generate documentation also for classes that inherit from graphene types.

Since the library is already making assumptions about the existence of Django (and other metaprogramming frameworks), I think it would just suffice to produce a warning about this situation: if the _meta attribute has no 'get_fields' method, warn the user that IF they are using Django, there might be something wrong. Maybe just catch the AttributeError.

System (please complete the following information):

  • pytkdocs version 0.15
  • graphene version 2.1.8
  • Python version: 3.8
  • OS: Linux 5.4.0-94-generic # 106-Ubuntu SMP

Workaround
I have a workaround, abusing setup_commands for the mkdocstrings "python" handler, in order to patch my Graphene objects. I assign a 'get_fields' method to graphene's BaseOptions class, which is instantiated as a '_meta' attribute for a lot of Graphene objects.

How to patch:
fix_mkdoc_for_graphene.py:

from graphene.types.base import BaseOptions


class DjangoMetaMock:
    def __init__(self, x):
        self.null = False
        self.blank = False
        self.help_text = False
        self.verbose_name = x


def patch():
    """ Patches any class that may be assigned to one of Graphene's type's _meta attribute.

    Mkdocstrings' pytkdocs cannot generate documentation for classes using Graphene's types. This is because pytkdocs assumes the existence of a 'get_fields' method on any '_meta' member of a class.
    """
    # Patch the 'BaseOptions' class.
    BaseOptions.get_fields = lambda x: [DjangoMetaMock(x)]

setup_commands in mkdoc.yml:

- mkdocstrings:
    default_handler: python
    handlers:
      python:
        setup_commands:
          - import fix_mkdoc_for_graphene
          - fix_mkdoc_for_graphene.patch()
@pawamoy
Copy link
Member

pawamoy commented Feb 3, 2022

Hi @jaspergreener, thank you very much for the detailed bug report, suggestions and workarounds.
The current Python handler uses pytkdocs to extract data, and pytkdocs is... not the best. I've been working on an much improved version, Griffe, and it should actually be available soon. I believe the new handler will fix this kind of issues, so I won't work on resolving this in pytkdocs for now. But I'll leave the issue open 🙂

@pawamoy
Copy link
Member

pawamoy commented Feb 7, 2022

This is likely solved by the new handler. Feedback appreciated!

@pawamoy pawamoy closed this as completed Feb 7, 2022
@jaspergreener
Copy link
Author

Very happy with the quick response! I'll give it a try and let you know in this issue :)

@jaspergreener
Copy link
Author

The new handler isn't working for me, unfortunately. It seems to fail on generating documentation for a class inheriting from a pydantic.BaseModel:

INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /home/user/PycharmProjects/greener-projects-be/docs/site
ERROR    -  Error reading page 'administration.md': <class '_ast.keyword'>
Traceback (most recent call last):
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocs/__main__.py", line 187, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocs/commands/build.py", line 292, in build
    _populate_page(file.page, config, files, dirty)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocs/commands/build.py", line 174, in _populate_page
    page.render(config, files)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocs/structure/pages.py", line 174, in render
    self.content = md.convert(self.markdown)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/markdown/core.py", line 264, in convert
    root = self.parser.parseDocument(self.lines).getroot()
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/markdown/blockparser.py", line 90, in parseDocument
    self.parseChunk(self.root, '\n'.join(lines))
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/markdown/blockparser.py", line 105, in parseChunk
    self.parseBlocks(parent, text.split('\n\n'))
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/markdown/blockparser.py", line 123, in parseBlocks
    if processor.run(parent, blocks) is not False:
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocstrings/extension.py", line 120, in run
    html, handler, data = self._process_block(identifier, block, heading_level)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocstrings/extension.py", line 185, in _process_block
    data: CollectorItem = handler.collector.collect(identifier, selection)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/mkdocstrings_handlers/python/collector.py", line 69, in collect
    loader.load_module(module_name)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/loader.py", line 131, in load_module
    top_module = self._load_module_path(package.name, package.path, submodules=submodules)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/loader.py", line 287, in _load_module_path
    self._load_submodules(module)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/loader.py", line 292, in _load_submodules
    self._load_submodule(module, subparts, subpath)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/loader.py", line 303, in _load_submodule
    member_parent[subparts[-1]] = self._load_module_path(
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/loader.py", line 283, in _load_module_path
    module = self._visit_module(code, module_name, module_path, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/loader.py", line 326, in _visit_module
    module = visit(
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 78, in visit
    return Visitor(
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 158, in get_module
    self.visit(top_node)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 169, in visit
    super().visit(node)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/base.py", line 19, in visit
    getattr(self, f"visit_{node.kind}", self.generic_visit)(node)  # type: ignore[attr-defined]
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 200, in visit_module
    self.generic_visit(node)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 182, in generic_visit
    self.visit(child)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 169, in visit
    super().visit(node)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/base.py", line 19, in visit
    getattr(self, f"visit_{node.kind}", self.generic_visit)(node)  # type: ignore[attr-defined]
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 236, in visit_classdef
    self.generic_visit(node)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 182, in generic_visit
    self.visit(child)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 169, in visit
    super().visit(node)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/base.py", line 19, in visit
    getattr(self, f"visit_{node.kind}", self.generic_visit)(node)  # type: ignore[attr-defined]
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/visitor.py", line 502, in visit_annassign
    self.handle_attribute(node, get_annotation(node.annotation, parent=self.current))
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 681, in get_annotation
    return _get_annotation(node, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 666, in _get_annotation
    return _node_annotation_map[type(node)](node, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 621, in _get_subscript_annotation
    subscript = _get_annotation(node.slice, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 666, in _get_annotation
    return _node_annotation_map[type(node)](node, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 648, in _get_index_annotation
    return _get_annotation(node.value, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 666, in _get_annotation
    return _node_annotation_map[type(node)](node, parent)
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 580, in _get_call_annotation
    kwargs = Expression(*[_get_annotation(kwarg, parent) for kwarg in node.keywords])
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 580, in <listcomp>
    kwargs = Expression(*[_get_annotation(kwarg, parent) for kwarg in node.keywords])
  File "/home/user/PycharmProjects/greener-projects-be/venv3.8/lib/python3.8/site-packages/griffe/agents/nodes.py", line 666, in _get_annotation
    return _node_annotation_map[type(node)](node, parent)
KeyError: <class '_ast.keyword'>

Process finished with exit code 1

Griffe tries to get an annotation for an AST Keyword, but has no annotation implementation for not keywords: the node annotation map does not contains type ast.keyword.

@pawamoy If you like, I could report this in a separate issue.

@pawamoy
Copy link
Member

pawamoy commented Feb 11, 2022 •
edited

@jaspergreener Yes please! On the griffe repository preferably 🙂 And thanks a lot for testing the new handler 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pawamoy @jaspergreener