-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extract attribute docstrings for FieldInfo.description
#6563
Merged
+521
−7
Merged
Changes from all commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
29bbf3e
WIP: Implement docs extraction
Viicos 792da19
Implement attribute docstring extractiob
Viicos d3e8402
Apply feedback
Viicos c9325d7
Implement `ast.NodeVisitor`
Viicos 1f81724
Add `use_attributes_docstring` config value
Viicos cab7a07
WIP: First tests
Viicos e46890c
Fix tests to cope with `inspect.getsource`
Viicos af4f340
Add config docstring
Viicos d3a4761
Support `TypedDict`
Viicos 3da7402
Walk back frames to get source code
Viicos 4f73618
Handle the case where `f_lineno` is not an `int`
Viicos 2102756
Fix usage of `getblock`
Viicos 8ae16c0
Improve class detection when walking up frames
Viicos 436e95a
Apply feedback and last fixes
Viicos 85e4c0f
Add test with `create_model`
Viicos d8cbc3b
WIP: Feedback and additional tests
Viicos f8c56a9
More tests and small code refactors
Viicos 44a4a73
Use same model names in tests
Viicos 43c1c82
Fix rebase
Viicos 626a19a
Use source lines from `frame` object
Viicos 359846b
Improve config docstring, apply additional feedback, fix infinite loop
Viicos c12f020
`pyright`
Viicos b9ca191
lint again
Viicos 05598d3
Remove unrelated file
Viicos a044bce
Fix handling of dataclasses with Python>=3.11
Viicos 23c8695
Fix return value
Viicos 70b5ec2
Parse the frame block using `ast`
Viicos 7d9d77a
Dedent fix
Viicos 0f1743b
Some optimizations
Viicos ab3dfc0
Comments reordering
Viicos d451205
Remove commented `breakpoint()`
Viicos 146f915
Add additional generic assertion
Viicos 23396b8
Fix failing test
Viicos 73f1a94
Apply minor changes and fixes
Viicos f3e84a6
Update to latest `pyright`
Viicos d9150c6
fix rebase
Viicos abba7d2
Use latest pyright
Viicos caa5cbe
Merge branch 'main' into docstrings-description
Viicos File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
"""Utilities related to attribute docstring extraction.""" | ||
from __future__ import annotations | ||
|
||
import ast | ||
import inspect | ||
import textwrap | ||
from typing import Any | ||
|
||
|
||
class DocstringVisitor(ast.NodeVisitor): | ||
def __init__(self) -> None: | ||
super().__init__() | ||
|
||
self.target: str | None = None | ||
self.attrs: dict[str, str] = {} | ||
self.previous_node_type: type[ast.AST] | None = None | ||
|
||
def visit(self, node: ast.AST) -> Any: | ||
node_result = super().visit(node) | ||
self.previous_node_type = type(node) | ||
return node_result | ||
|
||
def visit_AnnAssign(self, node: ast.AnnAssign) -> Any: | ||
if isinstance(node.target, ast.Name): | ||
self.target = node.target.id | ||
|
||
def visit_Expr(self, node: ast.Expr) -> Any: | ||
if isinstance(node.value, ast.Str) and self.previous_node_type is ast.AnnAssign: | ||
docstring = inspect.cleandoc(node.value.s) | ||
if self.target: | ||
self.attrs[self.target] = docstring | ||
self.target = None | ||
|
||
|
||
def _dedent_source_lines(source: list[str]) -> str: | ||
# Required for nested class definitions, e.g. in a function block | ||
dedent_source = textwrap.dedent(''.join(source)) | ||
if dedent_source.startswith((' ', '\t')): | ||
# We are in the case where there's a dedented (usually multiline) string | ||
# at a lower indentation level than the class itself. We wrap our class | ||
# in a function as a workaround. | ||
dedent_source = f'def dedent_workaround():\n{dedent_source}' | ||
return dedent_source | ||
|
||
|
||
def _extract_source_from_frame(cls: type[Any]) -> list[str] | None: | ||
frame = inspect.currentframe() | ||
|
||
while frame: | ||
if inspect.getmodule(frame) is inspect.getmodule(cls): | ||
lnum = frame.f_lineno | ||
try: | ||
lines, _ = inspect.findsource(frame) | ||
except OSError: | ||
# Source can't be retrieved (maybe because running in an interactive terminal), | ||
# we don't want to error here. | ||
pass | ||
else: | ||
block_lines = inspect.getblock(lines[lnum - 1 :]) | ||
dedent_source = _dedent_source_lines(block_lines) | ||
try: | ||
block_tree = ast.parse(dedent_source) | ||
except SyntaxError: | ||
pass | ||
else: | ||
stmt = block_tree.body[0] | ||
if isinstance(stmt, ast.FunctionDef) and stmt.name == 'dedent_workaround': | ||
# `_dedent_source_lines` wrapped the class around the workaround function | ||
stmt = stmt.body[0] | ||
if isinstance(stmt, ast.ClassDef) and stmt.name == cls.__name__: | ||
return block_lines | ||
|
||
frame = frame.f_back | ||
|
||
|
||
def extract_docstrings_from_cls(cls: type[Any], use_inspect: bool = False) -> dict[str, str]: | ||
"""Map model attributes and their corresponding docstring. | ||
|
||
Args: | ||
cls: The class of the Pydantic model to inspect. | ||
use_inspect: Whether to skip usage of frames to find the object and use | ||
the `inspect` module instead. | ||
|
||
Returns: | ||
A mapping containing attribute names and their corresponding docstring. | ||
Viicos marked this conversation as resolved.
Show resolved
Hide resolved
|
||
""" | ||
if use_inspect: | ||
# Might not work as expected if two classes have the same name in the same source file. | ||
Viicos marked this conversation as resolved.
Show resolved
Hide resolved
|
||
try: | ||
source, _ = inspect.getsourcelines(cls) | ||
except OSError: | ||
return {} | ||
else: | ||
source = _extract_source_from_frame(cls) | ||
|
||
if not source: | ||
return {} | ||
|
||
dedent_source = _dedent_source_lines(source) | ||
|
||
visitor = DocstringVisitor() | ||
visitor.visit(ast.parse(dedent_source)) | ||
return visitor.attrs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this work?
Personally I'm not convinced that using a visitor is clearer or better than your first implementation. Visiting every node in the tree is a waste, and I don't like having to think about the order in which the visit methods get called.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before applying the suggestion/reverting to the old solution, I'll let maintainers decide what they prefer.