Search: compact summaries for long code blocks and lists

[feasgal]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the problem doesn't occur with the mkdocs or readthedocs themes
• ... the problem persists when all overrides are removed, i.e. custom_dir, extra_javascript and extra_css
• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

The search results preview displays the whole <ol> or <ul>, regardless of where in the list the result appears. This makes for a lot of scrolling in the results. I first commented about this on #3053 in April, because I mistakenly thought I was getting the entire page in the result. Now that I've narrowed it down, I'm back with a new issue as requested.

Expected behaviour

Similar to results that appear in a plain paragraph, I would expect the search preview to display the <li> in which the search term appears. If the <li> includes multiple paragraphs, maybe even just the relevant <p>.

Actual behaviour

I have an ordered list, several of whose list items contain nested, unordered lists. See attached lorem ipsum file long_search_result.md

When served or built, if I search for TN2206, I get the entire <ol> in the preview, with its nested unordered lists, even though TN2206 only appears once in the <ol>, as the last word of the last <li> in the last <ul> in the last <ol>.

When served or built, if I search for cras, I get the entire <ol> in the preview, with its nested unordered lists, even though cras only appears once in the <ol>, in the first <li> of the top <ol>.

Steps to reproduce

1. Serve or build an MkDocs project containing long_search_result.md.
2. Search the served/built docs for TN2206.
3. Search the served/built docs for cras.

Package versions

• Python: Python 3.10.0
• MkDocs: mkdocs, version 1.3.0 from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mkdocs (Python 3.10)
• Material: Version: 8.4.0+insiders.4.21.1

Configuration

site_name: Long Search Results
site_url: "https://example.com/docs"

plugins:
    - search:
        separator: '[\s\-/\\]+'
    - offline

nav:
    - Example: long_search_result.md
    
theme:
 name: material


markdown_extensions:
   - smarty
   - admonition

System information

• Operating system: macOS Monterey Version 12.4
• Browser: Google Chrome Version 104.0.5112.101 (Official Build) (x86_64)

[squidfunk]

Thanks for reporting. This is actually not a bug, but definitely yields room for improvement. I'll check how we can improve search summaries of structured data (lists, code blocks) in the coming months!

[squidfunk]

I tinkered a little with the search, and following is an idea we can discuss – before shortening text, we would gain a lot if we just remove (or collapse) the list elements that contain no occurrences of a search term. Take for example this query:

Here are all elements that contain no matches (just for illustration, no real UI proposal):

Here is the list with only elements that contain matches:

The looks are still a little meh, because the collapse indicators take too much space. IMHO, it is critical to preserve list numbering, so bullet points would not have different numbers when following up to the page.

This could be extended to code blocks as well. I'm currently thinking about the next big iteration of the search plugin, and we maybe we should move away from making the whole result clickable, because this would allow to make search results completely interactive. We could collapse certain sections, expand them on click, even allow to copy from clipboard.

I'm just thinking out loud. I'll keep this issue updated when I make progress. Any feedback is appreciated.

[feasgal]

Yes, even just the above would help a lot.

Code blocks are definitely also a dramatic example in our specific use case...when they happen, which is nowhere near as frequently as the lists example. So even just trimming the display of results in lists, like you have above, would be much appreciated.

[TristisOris]

i was confused a bit, because your own blog say that search truncated to 320 symbols, but it not. https://squidfunk.github.io/mkdocs-material/blog/2021/09/13/search-better-faster-smaller/#search-previews

[squidfunk]

@TristisOris the old search truncated to 320 characters, the new (current) one doesn't:

This was due to the fact that search previews were truncated after a maximum of 320 characters, ...

[squidfunk]

Since this was asked multiple times, here's a mitigation for the overly large previews that reverts to the pre-v9 behavior of Material for MkDocs, truncating search after three lines (albeit you can change this value). Just add the following CSS:

.md-search-result .md-typeset {
  display: -webkit-box;
  -webkit-line-clamp: 3;
  -webkit-box-orient: vertical;
}

[TristisOris]

[Just add the following CSS]

This break some blocks and show empty info. And can't truncate lists, code, etc.

so best balanced option for now - use as is.

[squidfunk]

@TristisOris could you please elaborate?

[TristisOris]

@squidfunk

[squidfunk]

Thanks, give me some time to check how we can omit the behavior mentioned.

[ekoleda-codaio]

+1 to this issue. The search suggestions for my docs can be very long, to the point where it dramatically hurts the utility of the search functionality.

[squidfunk]

Understood, and actually one of the most important things on our roadmap for the new search implementation. Here's an updated version of the previously proposed temporary fix, that reverts the height of the search to the previous behavior:

.md-search-result .md-typeset {
  display: -webkit-box;
  -webkit-line-clamp: 3;
  -webkit-box-orient: vertical;
  max-height: 120px; /* <- adjust to fit */
}

[TristisOris]

yep, that works much better. thanks.

[squidfunk]

Please see the announcement in #6307.

[HarelM]

First and foremost, amazing product! I searched for something that I can use for our docs that is simple and powerfull and boy, this library is amazing!!

I'm using it in the following site:
https://maplibre.org/maplibre-gl-js/docs/

It works great except for one minor thing:
I have these html examples here:
https://maplibre.org/maplibre-gl-js/docs/examples/

I basically wrote a small script that converts our testing examples to docs so they can be seen in the site, with an iframe and a code block, pretty straight forward.

The problem is that if I search for one of the examples I get the full code in the search results, which doesn't allow to see other results, this is even worse on mobile device:

Here's my configuration:
https://github.com/maplibre/maplibre-gl-js/blob/main/mkdocs.yml

Thanks for all the amazing work done here!!!

I'll try the provided CSS fix, but I mainly wanted you to have an example of a production open source documentation project with this issue.

[HarelM]

The provided CSS did the trick, thanks!! 🙏

[facelessuser]

If you want to avoid certain code content, just apply the data-search-exclude attribute to exclude them. This can be done with attr_list extension or added directly to HTML elements.

[HarelM]

Thanks @facelessuser! I want the code to be shown, I just want it shorter.
Relevant PR was merged :-) maplibre/maplibre-gl-js#3400

[facelessuser]

Ah, I couldn't tell from the image.

[Akkadius]

We just upgraded our docs after 3 years of being users in our open source community - the search has always been amazing and we immediately noticed it functionally was worse in many ways.

Docs https://docs.eqemu.io/schema/characters/character_data/

I appreciated that we'd always see a short and succinct glimpse of where the keyword is on the document, now it often comes with huge blobs of text you need to scroll to even see the results for. The CSS workaround feels nicer but I often don't even see where the keyword is in the context of the findings to know whether I care for that search result wheres before I easily could.

Now, things in code blocks aren't even showing up in the search as they used to before.

Also - I appreciate all the work you've put into this from another open sourcer - I added to your Chipotle fund 😄

Am I able to at least revert the search functionality to how it was before until the new search is ironed out more ?

[squidfunk]

Thanks for supporting the project! 🚀

The CSS workaround feels nicer but I often don't even see where the keyword is in the context of the findings to know whether I care for that search result wheres before I easily could.

The CSS workaround should get you exactly back to the previous behavior. We are very aware of this issue and working very hard on improving the search experience, as I explained in #6307. The problems reported in this issue will be solved as part of the ground up rewrite. There's also a research preview with the first prototype of the new engine in #6372 – we need to collect more feedback on that to push it forward.

Now, things in code blocks aren't even showing up in the search as they used to before.

This might be related to the configuration of the search separator, and is likely something that can be fixed with configuration. You can create a minimal reproduction and ask on our discussion board for help ☺️

[Akkadius]

Thanks for supporting the project! 🚀

The CSS workaround feels nicer but I often don't even see where the keyword is in the context of the findings to know whether I care for that search result wheres before I easily could.

The CSS workaround should get you exactly back to the previous behavior. We are very aware of this issue and working very hard on improving the search experience, as I explained in #6307. The problems reported in this issue will be solved as part of the ground up rewrite. There's also a research preview with the first prototype of the new engine in #6372 – we need to collect more feedback on that to push it forward.

Now, things in code blocks aren't even showing up in the search as they used to before.

This might be related to the configuration of the search separator, and is likely something that can be fixed with configuration. You can create a minimal reproduction and ask on our discussion board for help ☺️

Thanks for the reply and all that you do. I can imagine all that you're going through to rewrite all of the components of the search from scratch and appreciate what you've likely had to do with insiders and sponsors to work through crazy edge / use cases getting docs to scale and probably spent more time on the problem than you'd like necessitating the rewrite.

I'm fine with workarounds or things that bring us to prior functionality for now until things settle, but like I said before even the CSS workaround just isn't it. When I search for "crystal" I get shortened results sure, that helps, but I don't see contextually within the search result where "crystal" is.

There are also searches that just don't show up at all anymore as I mentioned in my previous post - that's not helped by the CSS workaround as that's just a visual adjustment.

CSS Change

Is there a way I can revert to the old search for now until the dust settles? I personally never had any issues with it and it worked phenomenally well for our community

[squidfunk]

I'm fine with workarounds or things that bring us to prior functionality for now until things settle, but like I said before even the CSS workaround just isn't it. When I search for "crystal" I get shortened results sure, that helps, but I don't see contextually within the search result where "crystal" is.

That is the behavior of the old search. It only every showed the 200-300 characters of a section and cut off after that, even though the actual occurrence might be located in a subsequent part of the text.

Is there a way I can revert to the old search for now until the dust settles? I personally never had any issues with it and it worked phenomenally well for our community

If you feel that the old one worked better, you can downgrade to 8.5.10, but it's definitely not recommended. As said, you can open a discussion, so somebody can help you improve the results as it sounds they might be related to the separator as I wrote before, but you cannot just switch back to the previous search, because the plugin is not the only thing that needed to be changed – the application itself would need to be patched in several places.

[Akkadius]

It only every showed the 200-300 characters of a section and cut off after that, even though the actual occurrence might be located in a subsequent part of the text.

I'm going to be honest, it is surprising we replaced the current search wholesale without compact summaries as a minimum requirement. It is quite a bit of a UX regression when used to the previous experience where very quickly at a glance you had lightning fast search and could scan content very quickly as you change your search terms. Now you have to stop, mouse over, search and scroll through pages of content to grok what it is you're trying to find and it's not a great experience.

I understand the reason for moving to something new and the reasons for investing in a new search algorithm, but I would think we would feature parity before GA and at least feature flag until we were ready.

I don't have time to troubleshoot document search as I have a lot of work to do in my own open source community. I'm happy to provide basic info wherever it is helpful though. If it's helpful this is our doc site https://docs.eqemu.io/

I'm going to leave it in its current state until search gets sorted.

I think this is by far and away the most polished, incredible, clean, thoughtful, impressive markdown documentation based project out there today and have had almost zero complaints and more than happy to support this amazing project, this one makes me 🤨

[squidfunk]

I'm going to be honest, it is surprising we replaced the current search wholesale without compact summaries as a minimum requirement. It is quite a bit of a UX regression when used to the previous experience where very quickly at a glance you had lightning fast search and could scan content very quickly as you change your search terms. Now you have to stop, mouse over, search and scroll through pages of content to grok what it is you're trying to find and it's not a great experience.

The current implementation is actually more sophisticated than the previous one, as it understands paragraphs and will only show the first two paragraphs which have occurrences that match (e.g. check this query). The CSS workaround will still improve the situation over the previous implementation, because it will be more likely that an occurrence is shown as part of the result. Thus, please apply the CSS workaround for the time being and get the same behavior as before.

I don't have time to troubleshoot document search as I have a lot of work to do in my own open source community. I'm happy to provide basic info wherever it is helpful though.

I understand. However, we're only as good as the information we get, and as I said, I'm sure it can be solved. So, if you at some point manage to squeeze in a few minutes, you can open a discussion and somebody can improve the situation with you together. In the meantime, we'll be working on improving the default and trying to make it even more "out-of-the-box". Material for MkDocs provides one of the best search implementations without the need for a server that supports 67 languages. Is it perfect? Absolutely not, but we're working on moving it closer to perfect every day.

I'm sorry that the current implementation does not fit your requirements. I can only say that I'm working tirelessly on the new implementation and we will have something to test drive very soon. The decision to go down the "from scratch" route came up after several change requests came up, all related to search, which cannot be solved with the current architecture. Thus, we decided to stop monkey-patching Lunr.js and solve things the right way.

All of this takes time, but it is currently our highest priority. Thank you for your understanding.

[Akkadius]

I'm sorry that the current implementation does not fit your requirements. I can only say that I'm working tirelessly on the new implementation and we will have something to test drive very soon. The decision to go down the "from scratch" route came up after several change requests came up, all related to search, which cannot be solved with the current architecture. Thus, we decided to stop monkey-patching Lunr.js and solve things the right way.

I was being frank in my feedback but I hope you read my sincerest respect for you, your work and this project. It's outstanding. Open source is brutal and you my friend are doing great <3