Incompatibilities with search and Sphinx 5.x

[stsewd]

TL;DR: Sphinx is dropping jQuery sphinx-doc/sphinx#10028 and those changes break our custom server side search feature (SSS), and it also breaks the search page on our theme (with or without our SSS).

The first problems were reported at #8862.

• Seach.init is being called twice, duplicating some elements (will be fixed with Search: correctly remove Search.init on Sphinx >= 5.0 readthedocs-sphinx-ext#100)

• Our custom js that replaces Sphinx's default search with our SSS assumes that some attributes from the Search object are jQuery objects, but after the refactor done upstream, they will be HTML/Node object

  We need to check all usages of Sphinx.* and decide what to do.

  • readthedocs.org/readthedocs/core/static-src/core/js/doc-embed/search.js

    Line 209 in e7102a4

    Search.title.text(_('Search Results'));

    
    All calls to Sphinx.{element}.text() should be easy to fix, we can have a utility function that checks if the element is an jQuery object, something like

    const setText = function(node, text) {
      if (node.jquery) {
        node.text(text);
      } else {
        node.innerText = text;
      }
    };

  • readthedocs.org/readthedocs/core/static-src/core/js/doc-embed/search.js

    Lines 209 to 211 in e7102a4

    	Search.title.text(_('Search Results'));
    	Search.status.fadeIn(500);
    	});

    I think we can delete that call to .fadeIn, it's done at the end of the search operation, so at this stage the text from Sphinx.status is already shown. Actually, I think this was meant to be Sphinx.title.fadeIn.

  • readthedocs.org/readthedocs/core/static-src/core/js/doc-embed/search.js

    Lines 191 to 192 in e7102a4

    	Search.output.append(list_item);
    	list_item.slideDown(5);

    list_item is a jQuery object (we are making heavy use of jQuery to create that element), but now Search.output will be an HTML/Node object, we can do something like this:

    if (Search.output.jquery) {
      Search.output.append(list_item);
      list_item.slideDown(5);
    } else {
      Search.output.appendChild(list_item.get(0));
      // No slideDown for the non-jquery version.
    }

• If some expected elements aren't on the search page, the search operation will fail.

  https://github.com/sphinx-doc/sphinx/blob/3b01fbe2adf3077cad5c2cf345c6d000d429d7ac/sphinx/themes/basic/static/searchtools.js#L211-L213

  Our theme completely overrides the search page, and it doesn't inherit from Sphinx's default search page

  https://github.com/readthedocs/sphinx_rtd_theme/blob/d64dadf1ceec4f9ff6c1ca2d3ea4c3d0fdb0e8d2/sphinx_rtd_theme/search.html#L10

  When retrieving a non-existent element, jQuery would still return an object (kind of empty one, so method calls won't raise a null exception), but now getElementById will return null and raise an exception when trying to call a method on that value.

  We can fix this in our theme (by adding that element or by inheriting from sphinx's search page to avoid things like this in the future),
  or we can also send a patch to sphinx to not assume that that element will always be present (it's used in 2 places only, so it shouldn't be a problem) Searchtools: don't assume that all themes define some elements sphinx-doc/sphinx#10153.

Another approach

Another way to solve most of these problems is to stop relying on sphinx's Search object and create and display the elements by ourselves, the mkdocs implementation doesn't rely on any mkdocs internals to display the search results.
This way, if sphinx changes the internals of the Search object, we won't be affected.

A downside would be that the structure/classes of the containers wouldn't be dictated by sphinx anymore,
so if our structure is different from the one expected for that version of sphinx, the styles may look off if the theme doesn't play nice with our structure.

Related issues

• Searchtools: don't assume that all themes define some elements sphinx-doc/sphinx#10153
• Searchtools: fix null query on setIndex sphinx-doc/sphinx#10233
• doctools: fix hideSearchWords sphinx-doc/sphinx#10244
• doctools: fix shortcuts sphinx-doc/sphinx#10245

---

Those are the issues I have found so far, additionally Sphinx will soon be dropping jQuery too (and we make heavy use of jQuery in our injected footer and friends).

[pradyunsg]

FWIW, this also affects Sphinx 4 sites.

[stsewd]

@pradyunsg do you have a link to those projects? What are the problems present on 4.x?

[stsewd]

We have deployed the fixes, search is working as expected on sphinx's master docs https://www.sphinx-doc.org/en/master/search.html?q=test

For search to work on our theme we only need to wait for sphinx-doc/sphinx#10153 to get merged.