use official pydata sphinx theme #10971

bryevdv · 2021-02-28T06:28:21Z

issues: fixes Move Bokeh docs to the latest version of pydata-sphinx-theme #10685
release document entry (if new feature or API change)

This is an early stab at removing our custom vendored version of the "pydata sphinx theme" for the official package. Unfortunately, I would say the first results are a bit wider off the mark than I had wished for. I am hoping @jorisvandenbossche or @choldgraf might be able to help shed some light or offer some pointers. I've outlined the main issues so far

Issues that aren't just styling

sidebars do not float

In the pandas docs and our old docs, the sidebars float as the central area is scrolled, affording constant navigation menus. When I build with the latest 0.4.3 version of the pydata theme, both the left and right sidebars scroll off the page:

At this point we don't have any custom CSS or JS at all so I am not sure what could be interfering.

scroll spy is not functions

Potentially related to the above comment. The "scrollspy" functionality to highlight the current section in the view port is simply not working at all.

Probably just styling

index page too narrow

We want both sidebars to be removed on the index page. Otherwise, we get this:

This also breaks the images at the bottom:

Gallery page too narrow

Same story as above, we have not historically had sidebars on the gallery, because we want to fill all available space with gallery tiles:

Version selector is gone

We will need to figure out how to inject our custom page JS that adds our version selector to the top nav bar

External link fa icons off center

Releases page nav is badly formatted

Gallery detail pages badly formatted

sphinx/source/conf.py

jorisvandenbossche · 2021-02-28T21:35:01Z

@bryevdv cool, thanks for trying this out!

First, I would recommend to directly use the latest master version (you can still pin it to a specific commit to use a "stable" version) until the next release, since there are quite some changes since the last release that can be useful, especially regarding styling (CSS variables).

Eg those variables should make it easy to use the custom colors you were using before.

In the pandas docs and our old docs, the sidebars float as the central area is scrolled, affording constant navigation menus. When I build with the latest 0.4.3 version of the pydata theme, both the left and right sidebars scroll off the page:

That is strange, especially if you didn't include any custom adaptations for now, since many other sites are using the theme and don't have this problem.

Do you have a preview link to check out / investigate? (eg do you build / host the docs on PRs?)

bryevdv · 2021-02-28T21:42:56Z

@jorisvandenbossche Thanks for chiming in!

First, I would recommend to directly use the latest master version

I did seem to get a build error w/ master but there's alot in motion so I will try again and report back.

Do you have a preview link to check out / investigate? (eg do you build / host the docs on PRs?)

We can host test docs, but it's a manual process, I will upload a some later tonight when I am back in front of a keyboard, and leave a comment with the URL!

choldgraf · 2021-02-28T21:46:04Z

@bryevdv regarding the scrolling, I have found before that this kind of thing happens if bootstrap is loaded two times on the page. Could it be that you have bootstrap JS or CSS loaded both by the pydata theme and by some other thing?

bryevdv · 2021-02-28T21:56:54Z

@choldgraf I don't think so but I will also try to check on that later when I am back home. Thanks!

tcmetzger · 2021-02-28T22:36:12Z

We are using a Sphinx extension called sphinx-panels - that one also uses bootstrap and might lead to bootstrap being loaded twice.
@bryevdv you could try deactivating sphinx_panels for testing, or just include panels_add_bootstrap_css = False in conf.py (as described here: https://sphinx-panels.readthedocs.io/en/latest/#sphinx-configuration)

Edit: the broken images at the bottom of the landing page you mention in your initial comment are also rendered inside boxes created with sphinx-panels, so there might indeed be some connection here.

choldgraf · 2021-03-01T00:38:29Z

@tcmetzger ah yep - that's the same challenge that I have had with sphinx-panels and adding that config value fixes it 👍

bryevdv · 2021-03-01T02:23:16Z

Setting panels_add_bootstrap_css = False solved the sidebar scrolling issues, thanks all!

bryevdv · 2021-03-01T18:23:56Z

First, I would recommend to directly use the latest master version

@jorisvandenbossche one more question about the above. We install the theme in CI with a pip: stanza in a conda env file, is it possible to install from a repo commit using an env file?

choldgraf · 2021-03-01T18:26:15Z

@bryevdv i think so, try following a similar pattern to this one:

https://github.com/open-contracting/standard/blob/f7a466dd9265bb7ae6ecc17168b3de5f00496ddb/common-requirements.txt#L7

bryevdv · 2021-03-01T19:04:20Z

ref: https://stackoverflow.com/questions/19042389/conda-installing-upgrading-directly-from-github

jorisvandenbossche · 2021-03-01T21:27:26Z

Yes, I am using this in pandas as well: https://github.com/pandas-dev/pandas/blob/9da3ca23850332e52d19250abbdcb7c482dc5508/environment.yml#L116 (not sure the "egg" part is needed, for pandas it seems to work without)

tcmetzger · 2021-03-06T03:59:48Z

I finally had time to locally build this branch and take a look around - it looks very promising, I'm really looking forward to when we can use this with the docs!

Two thoughts at first glance:

We'll probably have to fix something with the bokeh-plot extension. For example, on /user_guide/layout.html#single-object I get quite some overlaps:

Also, it would be nice to have the right side bar menu in the reference guide as well (for example an entry for each class in a module).

All in all, this looks very promising!

tcmetzger · 2021-03-06T04:59:14Z

I did some experimenting and just committed two of those experiments (feel free to revert any of those, of course):

I added some custom CSS back in (that got deleted with the old theme). This will definitely need some more fine-tuning, but it brings back some of the familiar colors - and fixes the gallery thumbnails on the start page, for example:

I also added our logo and a favicon, just to feel a little more "at home" 😀

Two questions about that, though:

As far as I can see, the html_logo has to be a relative path to conf.py - is there an easy way to use an absolute path again (such as https://static.bokeh.org/logos/logotype.svg)? And the alt text for the logo is just "logo" - is there a way to customize that as well (e.g. "Bokeh")?
There is only one config varable for a favicon - is there a simple way to define several favicon sizes rel="apple-touch-icon" sizes="180x180", rel="icon" type="image/png" sizes="32x32", etc)?

The OpenGraph extension seems to be working well though, all the metadata seems to be right where it should!

bryevdv · 2021-03-06T05:22:17Z

As far as I can see, the html_logo has to be a relative path to conf.py

The next (unreleased) version will has an option for a remote logo URL

There is only one config varable for a favicon

That I don't know, maybe @jorisvandenbossche or @choldgraf can comment? If nothing else I think we an override/customize the page template (without having to have a whole new theme or vendor anything)

choldgraf · 2021-03-06T15:56:34Z

Interesting point about the favicon, the pydata theme doesn’t do anything special with favicons, but I think that this would be a nice improvement.

jorisvandenbossche · 2021-03-08T07:48:14Z

And the alt text for the logo is just "logo" - is there a way to customize that as well (e.g. "Bokeh")?

That's currently hard-coded, I see: https://github.com/pydata/pydata-sphinx-theme/blob/93b5115834435c995138da4be9ad7bbcfbca8127/pydata_sphinx_theme/docs-navbar.html#L16
But that's certainly something we could make configurable if desired.

For the favicon size, that's indeed standard sphinx behaviour at the moment (that's inserted right now by inheriting from the basic sphinx template). There seems to be an issue about it over there: sphinx-doc/sphinx#7600
But short term, that could indeed something to enable with the theme itself.

Feel free to open issues/PRs about this at https://github.com/pydata/pydata-sphinx-theme!

jorisvandenbossche · 2021-03-08T07:52:26Z

it brings back some of the familiar colors

See also https://github.com/pydata/pydata-sphinx-theme/blob/master/pydata_sphinx_theme/static/css/theme.css for an overview of the available variables. For example, to get the different flavor of purple as currently used in the bokeh docs for highlighting the active title in the right TOC sidebar, that should be possible by setting a single variable right now (color-active-navigation)

bryevdv · 2021-03-08T17:59:41Z

There's actually quite a lot of extra header material in our other sites:

<link rel="apple-touch-icon" sizes="180x180" href="https://static.bokeh.org/favicon/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="https://static.bokeh.org/favicon/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="https://static.bokeh.org/favicon/favicon-16x16.png">
<link rel="manifest" href="https://static.bokeh.org/favicon/site.webmanifest">
<link rel="mask-icon" href="https://static.bokeh.org/favicon/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
<meta property="og:title" content="Bokeh" />
<meta property="og:type" content="website" />
<meta property="og:url" content="https://bokeh.org/" />
<meta property="og:description" content="Bokeh is a Python-based visualization library, capable of building plots from simple charts to interactive dashboards.">
<meta name="image" property="og:image" content="http://static.bokeh.org/og/logotype-on-hex.png">
<meta property="og:image:alt" content="Bokeh logo on hex tiles" />
<meta name="twitter:card" content="summary_large_image" />
<meta property="twitter:site" content="@bokeh" />

Which is why I thought it might be simpler just to extend the page template. IIRC is is possible to point to a new template that inherits from an existing one without creating a whole new theme? (Or am I mis-remembering that)

Edit: I guess the OG elts might come from the og extension (@tcmetzger would know) but AFAIK the rest was hand applied in the vendored version

tcmetzger · 2021-03-08T19:19:41Z

@bryevdv Yes, all the og (and twitter) meta tags are generated by the OpenGraph extension. As far as I can see, this extension works well with the new theme.

bryevdv · 2021-03-12T02:21:12Z

@jorisvandenbossche trying out new 0.5 version and getting exceptions:

  File "/Users/bryan/anaconda/envs/dev/lib/python3.8/site-packages/pydata_sphinx_theme/__init__.py", line 110, in generate_toc_html
    add_header_level_recursive(soup.find("ul"), 1)
  File "/Users/bryan/anaconda/envs/dev/lib/python3.8/site-packages/pydata_sphinx_theme/__init__.py", line 103, in add_header_level_recursive
    ul["class"] = ul.get("class", []) + ["visible"]
AttributeError: 'NoneType' object has no attribute 'get'

New version does not like files that do not have headers, it seems, e.g. just:

.. toctree::
   :maxdepth: 2
   :glob:

   property/*

Is this something for us to adjust to or should I file a bug report?

bryevdv · 2021-03-12T02:27:22Z

@jorisvandenbossche also I only just realized that I think I have misunderstood what

html_theme_options = {
    "logo_link": "<other page or external link>"
}

is for. I thought it was a way to link to a remote logo file, but I think that's incorrect and that it only controls where the logo links to. Is there currently any way to use remote URLs for logos? Our logos are all on static.bokeh.org

bryevdv · 2021-03-12T02:48:19Z

@tcmetzger I have pushed some fixups to build with the latest release of the theme. I also included a template override for docs-navbar.html to load our SVG logo directly from the static site:

In this case, I had to wholesale override the jinja template. But if we want to make mods to other templates and they have relevant {% block %} sections, then we can probably get away with merely extending the existing theme template rather than replacing it. In general I am fine with customizations that are confined to the custom templates dir. My main goal is to delete all of our python code for the custom theme. For reference the templates that can be overriden are here:

https://github.com/pydata/pydata-sphinx-theme/tree/master/pydata_sphinx_theme

Edit: I'd also love to push any additional customization options back upstream and reduce the need for template customizations over time.

bryevdv · 2021-03-12T04:05:25Z

@tcmetzger I've pushed some more changes to add our custom footer that matches our other web properties (sort of... still WIP) and also added a custom.css that can be used to style it or anything else.

cc @mattpap we could probably use your input on the sizing mode example in the the layout chapter at some point.

tcmetzger · 2021-03-27T21:13:07Z

@bryevdv I committed some updates to fix the vertical spacing and to also add the favicons back in (with layout.html)

tcmetzger · 2021-03-28T20:07:43Z

@jorisvandenbossche Following up on adding the favicons to the theme for this pr, I have opened pydata/pydata-sphinx-theme#359 to add support for customized favicons to the theme itself.

bryevdv · 2021-03-28T21:02:19Z

@tcmetzger what is the minimum remaining work you want to see done on this before merging (i.e. if we wanted to do more work before 2.4 but were not able to, it would still be "ok")?

tcmetzger · 2021-03-28T22:04:40Z

@bryevdv I think we'll need to figure out the issue with removing the sidebars for gallery, gallery/**, and index. And then maybe just spend some time clicking and searching around to see if everything words. Looks really good so far, though!

Less pressing but nice to have would be going through all the colors and bring them into the Bokeh color scheme (e.g. headline text color, or colors for things like .. note::).

tcmetzger · 2021-03-31T22:26:48Z

@bryevdv My most recent commit makes the index and all gallery pages full width again:

In conf.py we needed to provide a path, not just a filename ("docs/gallery/**" instead of "gallery/**") to turn off sidebars. But I also changed layout.html to make things truely full width.

tcmetzger · 2021-04-02T04:34:05Z

@bryevdv I customized the theme's colors in custom.css to match the Bokeh palette and colors from the current docs. I'm pretty happy with how it looks now!

Could you build and deploy this branch somewhere (https://docs.bokeh.org/en/test/ ?) so that we could do some more testing?

bryevdv · 2021-04-04T00:17:26Z

@tcmetzger I have deployed to http://docs.bokeh.org/en/test using 2.3.1dev1 (there are no 2.4 dev builds yet)

bryevdv · 2021-04-04T00:22:53Z

Things look really good to me FWIW, I would be 👍 to merge and make folllow-on issues

bryevdv · 2021-04-04T00:44:29Z

@tcmetzger I pushed a small change to the ref guide model rendering that adds a section header above each model. This gets the models into the right nav menu:

Two comments:

I don't love the actual inline extra section header. Maybe we can get rid of it, or maybe it's not so bad.
This does not yet help things like Figure that has a a gazillion methods. That will required a different approach

tcmetzger · 2021-04-04T02:34:33Z

@bryevdv Thanks, that looks great! I'm OK with having a headline and the inline headline - this adds some good structure, and having the right hand navigation ist very helpful. Yes, we should probably improve this for things like Figure, but that could indeed be a follow up.

bryevdv · 2021-04-04T03:24:25Z

FYI I think we will have to use something like autoclasstoc for now, because creating an actual TOC for methods is still an open issue in sphinx. I have a POC but adding it is currently slow page load (esp bokeh.plotting) with 2.3 and also needs some other tweaks, so definitely a follow on.

@tcmetzger if you agree this can be merged as-is please go ahead and merges at your earliest

bryevdv · 2021-04-04T19:16:30Z

@tcmetzger Supposedly the CI issue is fixed but we need to kick off a build to test it, and I have some ref guide improvements I am itching to start. I am going to go ahead and merge this now and we can make follow-on issues as needed. Some things I know offhand:

The releases page is pretty ugly
- the RHS navbar should probably only go one-level deep (versions) on this one page
- The collapsible code blocks for hashes seems to not be collapsed/-ible at all any more

bryevdv · 2021-04-04T23:10:02Z

@choldgraf @jorisvandenbossche @tcmetzger anyone have any ideas for quick workarounds to limit the max TOC depth just on a single page? The new default of 2 that is great in most places is not great for the releases page:

I tried adding e.g. :maxepth: 1 to the page but it seemed to have no effect

tcmetzger · 2021-04-05T03:14:55Z

@bryevdv Is this behavior maybe linked to sphinx-doc/sphinx#6033? @jorisvandenbossche opened this issue, so he might know best.

A (not very elegant) way to solve this could be to include some file wide metadata at the top of the releases page to indicate that we want only a one-level TOC (e.g. :manual_toc_level: 1) and then check this value in one of our custom html files. If a value 1 for manual_toc_level is detected, we would use some CSS (.toc-h3 {display: none;}) to deactivate the second TOC level.

jorisvandenbossche · 2021-04-06T08:22:38Z

Wow, the result now looks really great!
(I am going to steal some ideas (eg the wider homepage) for pandas I think ;))

We should also try to upstream some changes to avoid you need to customize the full layout.html template.

anyone have any ideas for quick workarounds to limit the max TOC depth just on a single page?

Based on a quick experiment with our demo docs, it seems that the :tocdepth: (what @tcmetzger referenced) actually works. But, with the caveat that the number refers to the "global" level (full site document tree), and not the local page (making it more difficult to find the correct number to specify). So if you want to limit the number of levels on the releases page to 1, and the page itself is 1 level down (only the navbar entry, no sidebar), you would need something like :tocdepth: 2. Except that it seems to start to count at 0, so I would try a :tocdepth: 1 maybe.

Now, the :tocdepth: is a sphinx file metadata value, and it actually trims the TOC (so it also won't expand to more levels when scrolling down the page). While our theme-specific option show_toc_level only determines the visibility.
So if you want to change the default displayed level for a single page (without trimming the TOC itself, so it can still expand when scrolling), then we could indeed add a custom file metadata field to the theme, as suggested by @tcmetzger.

jorisvandenbossche

I quickly went through and add some comments / questions.

I think one edits you made in the layout.html was to support the wide page, right? (beeeb63) In the base theme, you can disable the sidebar, but then we still include some dummy (narrower) sidebar, which you removed I think? And in addition, you also removed the TOC right sidebar in those cases.
I think that would be something useful to have in the base theme as well, so we should think about how we could include that.

For the right TOC, I suppose we can do something as the left sidebar and make it smaller / remove it altogether if it's not present.

BTW, to give you an update in case you haven't seen: with the next version of the theme, you will also be able to add captions in the left sidebar + it now can collapse/expand subsections (see eg https://pydata-sphinx-theme.readthedocs.io/en/latest/demo/subpages/subpage1.html), in case you would want to make use of it.