Feature request: Make the search index pipeline configurable

[wilhelmer]

With the release of lunr.py 0.6.0, the lunr pipeline is now configurable. See my feature request there.

This means you can now generate a pre-built search index with, e.g., disabled stemming.

It would be nice to integrate this functionality in the search plugin, so we can modify the pipeline when using prebuilt_index, as outlined here:

- search:
    prebuild_index: python
    pipeline:
        - stemmer: false
        - trimmer: true
        - stopWordFilter: true

Currently, themes that modify the pipeline can only do this when the index is not pre-built, leading to inconsistent search behavior depending on whether prebuilt_index is enabled or not.

[squidfunk]

The syntax could be simplified to:

- search:
    prebuild_index: python
    pipeline:
        - stemmer
        - trimmer
        - stopWordFilter

If a transform should be omitted, it could just be dropped from search.pipeline, so IMHO there's no need for booleans. That's at least how we solve it over at Material. Also, the order in which the transforms are defined should be honored.

[wilhelmer]

Agreed.

[waylan]

I'm not opposed to the idea. A PR is welcome. That said, please keep in mind the discussion in yeraydiazdiaz/lunr.py#94.

[waylan]

Please see #2402 where we are beginning to back out support for lunr.py altogether. That said, I understand that some users will still want this. However, it should be relatively easy to implement as a third party plugin which subclasses the existing plugin. Presumably such a plugin would subclass mkdocs.contrib.search.search-index.SearchIndex and override the generate_search_index method and have it call lunr.py. Then a sublass of mkdocs.contrib.search.SearchPlugin would override the on_pre_build method to use the subclass of SearchIndex. Of course, the config handling would need to be overriden in the subclass also to include any of the custom config options for the pipeline etc.

In the end, most of the search implementation would still be done by the builtin plugin. Only the prebuilt index would be handled by the wrapper plugin. And that gives both lunr and the plugin the freedom to not be tied to MkDocs slower development cycle as well as the freedom to add any arbitrary options that may not be needed by MkDocs userbase at large..

[wilhelmer]

Doesn't the node-based method of pre-building the index have the same problem? We can't configure the pipeline there either.

[waylan]

Doesn't the node-based method of pre-building the index have the same problem? We can't configure the pipeline there either.

Yes, that's correct. However, when search was removed from MkDocs' core and refactored into a plugin, the intention was to simply replicate the basic behavior of the preexisting feature as a demonstration that the new Plugin API worked. However, the hope has always been that a third party search plugin would be created which added additional features and flexibility. We have never intended for the builtin search plugin to be the end-all be-all search solution. Therefore, if anyone wants to configure the pipeline using the node-based method, then that would need to be developed as a third-party plugin as well.

In the end, what I'm saying is that we don't intend to add this feature into MKDocs directly. We would prefer that this be implemented as a third-party plugin. While a plugin could be minimal and subclass the builtin plugin to add on a few features, I think an ideal solution would be for a third party plugin to provide a complete search solution that could become the standard way for users to get search in MkDocs.

[squidfunk]

This feature is now implemented in Material for MkDocs' own built-in search plugin and is already part of the v9 beta:

- search:
    pipeline:
      - stemmer
      - trimmer
      - stopWordFilter

[pawamoy]

Nowadays (2024) lunr.js is unmaintained (last commit 2020). While we welcome enhancements to existing themes, we're also discussing creating a new, better default theme, which would probably use something else than lunr.js. @squidfunk also stated that he and his team are working on a standalone search plugin that is theme-agnostic and will bring such functionality, see the discussion starting here #3560 (comment) (and this is already supported in the search plugin of Material for MkDocs).

For these reasons, and the ones listed by @waylan, we'll close this as not planned! If someone wants to work on this anyway, we can re-open the issue.