Is there a way to automatically exclude files without docstring?

[LuigiCerone]

Hello, is it possible to automatically exclude Python files without docstrings in it?

[pawamoy]

Hello! Yes, there's a way if you use the automatic reference recipe, though note that it will make collection happen twice: once to filter out modules without any docstrings, and a second time when actually rendering modules to HTML.

In your script that generates the reference pages, collect data for your package and check if each module has any docstrings in it. Something like this (not tested):

from pathlib import Path

import griffe
import mkdocs_gen_files

nav = mkdocs_gen_files.Nav()
data = griffe.load("your_package")
for path in sorted(Path("src").rglob("*.py")):
    module_path = path.relative_to("src").with_suffix("")
    doc_path = path.relative_to("src").with_suffix(".md")
    full_doc_path = Path("reference", doc_path)

    mod_parts = module_path.parts[1:]
    mod_parts[-1] = mod_parts[-1][:-3]  # remove .py
    if not data[mod_parts].has_docstrings:
        continue

    # rest of the recipe...

[quantfreedom]

@pawamoy I keep getting a key error on the mod_parts

can you explain what the mod parts is supposed to do because you say it is to remove the .py but you are already doing that in the module_path = part

can you give an example of what you are supposed to modify at mod_parts and then what are you supposed to modify and save as mod_parts[-1] =

[pawamoy]

Right, module_path already removes the suffix, so the line mod_parts[-1] = mod_parts[-1][:-3] is not necessary.

path == Path("src/package/sub/module.py")
module_path == Path("package/sub/module")
mod_parts == ["sub", "module"]
# then data[mod_parts] gives you the Griffe object, and we check if it has docstrings

[quantfreedom]

@pawamoy so now i am running into an error where every time it is reading any of my init files it says that they have doc strings ... the .has_docstrings is returning true even when they don't have doc strings ... but other files that aren't init files that don't have doc strings the has_docstrings returns false

so it is working on non init files ... am i doing something wrong or is it the griffe code?

[pawamoy]

has_docstrings (plural) searches for docstrings recursively. To only check the object and not its members, you can use has_docstring (singular) :)

[quantfreedom]

@pawamoy i think you are answering my old question that i deleted ... do you mind taking a look at the new updated question about the init files always returning True even when there is no doc strings in the file

because no matter if i use has_docstrings or has_docstring i get a true for strings and a false for string when looking at init files

[pawamoy]

That's expected behavior: your module does not have a module docstring, so has_docstring returns False. However the module has a function that has a docstring, so has_docstrings returns True.

[quantfreedom]

@pawamoy ok but then how does that explain this where there are no docstrings or functions or not even a line of code in this init file ... but still return true

So I'm still wondering why does this work correctly with regular python files that are not init files ...

#547 (comment)

[pawamoy]

Becaus has_docstrings also searches for docstrings in submodules. From the point of view of your users, quantfreedom/__init__.py (the file) is quantfreedom (the package). The user can import other things from below this package (in submodules), like for example quantfreedom.whatever.MyClass.

[quantfreedom]

@pawamoy so I'm guessing that means that there isn't a function to just check the init file and that I would have to make some kind of checker myself ... because there will be times where some of those init files have functions that have docstrings and some that have nothing and some that are just a docstring

[pawamoy]

You could simply iterate on members except submodules and check if they have docstrings:

for member in module.members.values():
    if not member.is_alias and not member.is_module and member.has_docstrings:
        # your logic

[quantfreedom]

@pawamoy so i have now removed everything from the init file and the docstrings is still returning true ... not even a line of code in the init file and still returns true

this is also in reference to my comment i made earlier #547 (comment)

[pawamoy]

I'm not sure what your use-case is, so I'll just give you some code, but this might not be the right approach to what you're trying to achieve.

for path in sorted(src.rglob("*.py")):
    if "_github" in str(path):
        continue

    module_path = path.relative_to(src.parent).with_suffix("")

    parts = tuple(module_path.parts)

    if parts[-1] == "__init__":
        parts = parts[:-1]
        init_module = data[parts[1:]] if len(parts) > 1 else data
        has_docstrings = init_module.has_docstring or any(
            member.has_docstrings for member in init_module.members.values()
            if not (member.is_alias or member.is_module)
        ) 
    else:
        has_docstrings = data[parts[1:]].has_docstrings

    if not has_docstrings:
        continue

    doc_path = path.relative_to(src.parent).with_suffix(".md")

    nav[parts] = doc_path.as_posix()
    with mkdocs_gen_files.open(doc_path, "w") as fd:
        ident = ".".join(parts)
        fd.write(f"::: {ident}")

    mkdocs_gen_files.set_edit_path(doc_path, Path("../") / path)

[quantfreedom]

All i want to do is check the init.py file and see if that individual file has any doc strings in it ... but it seems like there is a ton of miscommunication going on because the for loop you sent me loops through a bunch of different files checking if things are modules or alias or not and all i am looking to do is

in the if statement ... if parts-1 == init then i just want to check that file and that file only ... i want to read that exact file and see if there is any functions or any parts anywhere in the file that have doc strings in that file and that file only

i feel like there shouldn't have to be any for loops or anything because we know that we are looking at an init file

but it seems like that is impossible or there is just so much miscommunication that we don't understand each other unfortunately

is what i am saying making sense as far as what i want to do ...

i am not sure why i need to check members or check anything after we know we are looking at an init file

we know we have an init file and that should be it ...

i do understand what you mean about "If you have submodules in package, like package/submodule.py, then these submodules are considered members of the init module: from package import submodule, or from package.submodule import thing."

and i do understand about the has_docstrings checking sub modules and i understand that has_docstring checks if the file is basically just a doc string file

but it seems like the major problem here is that we can't just only check the init file as a file ... for some reason we can't just look at that file and that file only with something like init_has_docstrings ... and again i understand that graffie looks at init files as packages ... but why is it that we can't just look at the init file as the file itself

i know that it isn't a submodule but why can't we just look at the init file as a submodule

[pawamoy]

I'm not sure what you don't understand. I've given you a code snippet that does exactly what you are asking: checking if there docstrings in an __init__.py module without checking submodules.

    if parts[-1] == "__init__":
        # We're currently handling an __init__.py file.
        # Remove "__init__" suffix for accessing Griffe object in data.
        parts = parts[:-1]

        # Get Griffe object. If there's only one item in `parts`,
        # it's the package name, so it's simply `data` (which is the package collected by Griffe).
        # If there are more parts, we access the object by removing the first part, which is the package name,
        # because `data` is already the package.
        init_module = data[parts[1:]] if len(parts) > 1 else data

        # We consider the __init__ module "has docstrings" if:
        # - it has a module docstring
        # - or any of its members has a docstring
        # (attributes, functions, classes, but not modules or objects imported from other modules)
        has_docstrings = init_module.has_docstring or any(
            member.has_docstrings for member in init_module.members.values()
            if not (member.is_alias or member.is_module)
        ) 
    else:
        # If it's another module, simply use `has_docstrings` as a non-init module cannot have submodules anyway.
        has_docstrings = data[parts[1:]].has_docstrings

    # If we didn't find docstrings, continue to the next module file.
    if not has_docstrings:
        continue

As to why we can't just look at the __init__.py fiile as a submodule, is because Griffe gives you the view of a user of your API. Users never have to interact with __init__.py files explicitly, i.e. they never do from quantfreedom.__init__ import thing. So this technical detail (package/__init__.py is the contents of the package Python module) is hidden from Griffe's own API. In Griffe's API, there are only modules, which have members that can be attributes, functions, classes, or other modules.

[quantfreedom]

Thank you for adding the comments ... now i understand that member can be basically anything within the file or any modules or submodules for the package ... as i was debugging, members were only showing up as files .... but just now i added a function to the init file and then did some debugging and saw that the member can also be a function ... so the reasons behind the for loop makes sense now

the other part was the griffe giving us a view of the api instead of actually getting the file ... so i guess what i am wondering ... would there be any way to add something to griffe lib to make it view the init file differently like as if it was a submodule ?

[pawamoy]

You're welcome, I'm glad this helped you understand my previous code snippets.

would there be any way to add something to griffe lib to make it view the init file differently like as if it was a submodule ?

What is asked here is not very clear, but given the previous context my initial answer would be no: the current API was made like this by design, and I believe it is the right design for Griffe's purposes. Again, I'm not sure why you want to check if an __init__.py file has any docstring (I'm guessing you want to build a somewhat different navigation for your API reference than what is suggested in the official recipe) so feel free to create a new discussion where you expose your use-case in details.