Sage and Sphinx 2.0

[sedimentation-fault]

The constraint

<dev-python/sphinx-2.0.0

collides with the requirement

>=dev-python/sphinx-2.0

from dev-python/sphinxcontrib-qthelp - and dev-python/sphinxcontrib-qthelp is required by dev-python/pplpy, which in turn is required by the sage-9.0-r1 ebuild!

Now, maybe you say "I never needed those packages to build sage!". Indeed, neither did I - at least I had never even noticed their existence. Perhaps it is due to changes in the portage tree. Or to Python 2 problems. Or to my insistence on installing Sage for both Python 3.6 and 3.7. Or...or...

You choose. Fact is, you too will soon come across such a situation - the moment you need dev-python/sphinxcontrib-qthelp, for some of your packages, or for the fun of it. Personally, I find it unacceptable that a documentation package is prohibiting the use of a software - so I decided to put my finger in the wound and see WHY.

Here is what I have found out so far:

You need a few amendments to the ebuild - but nothing really dramatic:

The DEPEND list must look like this near the end:

...
    >=sci-libs/libhomfly-1.0.1
    sci-libs/libbraiding
    bliss? ( >=sci-libs/bliss-0.73 )
    >=dev-python/sphinx-1.8.5[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-applehelp-1.0.1[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-serializinghtml-1.1.3[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-htmlhelp-1.0.2[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-qthelp-1.0.2[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-devhelp-1.0.1[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-jsmath-1.0.1[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-websupport-1.1.0[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-spelling-4.3.0[${PYTHON_USEDEP}]
    >=dev-python/sphinxcontrib-programoutput-0.15[${PYTHON_USEDEP}]
    >=dev-python/jupyter_client-5.3.4[${PYTHON_USEDEP}]
    "

Notably I have thrown away the constraint <2.0 for sphinx and added all those sphinxcontrib-XXXXXX packages. I am sure that you need

dev-python/sphinxcontrib-applehelp-1.0.1
dev-python/sphinxcontrib-serializinghtml-1.1.3
dev-python/sphinxcontrib-htmlhelp-1.0.2
dev-python/sphinxcontrib-qthelp-1.0.2

because I've had sphinx errors telling me that module XXX is missing, for XXX=applehelp, serializinghtml, htmlhelp, qthelp. I can imagine that you need also programoutput and jsmath, because a documentation like that of Sage will want to show program code and math formulas. But I didn't specifically check, because at some point I got tired of re-building and waiting for an hour or so...Same is true for spelling and websupport. If you want a minimal set, you will have to check yourself if you can throw them out.

Your BDEPEND must look like this:

BDEPEND="app-portage/gentoolkit
    doc-html? ( sci-chemistry/sage-jmol-bin )
    doc-pdf? ( sci-chemistry/sage-jmol-bin )
    doc-pdf? ( app-text/texlive[extra,${L10N_USEDEP}] )"

i.e. I added dependencies on jmol, due to #579 .

The above is almost all that you need to get sage-9.0 built for both Python 3.6 and 3.7 with the USE flags

>=sci-mathematics/sage-9.0 -doc-pdf-bin doc-pdf doc-html -doc-html-bin

in your package.use.

Then what's the problem? The problem is one word above: ALMOST. :-)

Everything builds, the code and all the documentation...up to the references! 🤦‍♂️

The sphinx error is this:

# Sphinx version: 2.0.1
# Python version: 3.7.6 (CPython)
# Docutils version: 0.15.2 release
# Jinja2 version: 2.11.1
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/sphinx/cmd/build.py", line 283, in build_main
    args.tags, args.verbosity, args.jobs, args.keep_going)
  File "/usr/lib/python3.7/site-packages/sphinx/application.py", line 260, in __init__
    self._init_builder()
  File "/usr/lib/python3.7/site-packages/sphinx/application.py", line 321, in _init_builder
    self.emit('builder-inited')
  File "/usr/lib/python3.7/site-packages/sphinx/application.py", line 440, in emit
    return self.events.emit(event, self, *args)
  File "/usr/lib/python3.7/site-packages/sphinx/events.py", line 76, in emit
    results.append(callback(*args))
  File "/usr/lib/python3.7/site-packages/sphinx/ext/intersphinx.py", line 207, in load_mappings
    for key, (name, (uri, invs)) in app.config.intersphinx_mapping.items():
ValueError: too many values to unpack (expected 2)

but I doubt this is an error IN sphinx. The specific part of documentation generation where it occurs is:

...
[reference]     semirings: 0 todos, 3 index, 0 citations, 2 modules
[reference]     sets: 4 todos, 18 index, 0 citations, 17 modules
[reference]     stats: 0 todos, 11 index, 0 citations, 10 modules
[reference]     structure: 7 todos, 33 index, 0 citations, 32 modules
[reference]     tensor_free_modules: 2 todos, 19 index, 0 citations, 15 modules
[reference]     valuations: 1 todos, 14 index, 0 citations, 13 modules
[reference] ... done (489 todos, 2101 index, 1509 citations, 2051 modules)
[reference] preparing documents... skipping loading of indexes... done
[reference] The inventory files are in ../../build_doc/inventory/en/reference.
Build finished. The built documents can be found in /XXXXXX/portage/sci-mathematics/sage-9.0-r1/work/build_doc/inventory/en/reference
Building reference manual, second pass.

/XXXXXX/portage/sci-mathematics/sage-9.0-r1/work/sage-9.0/src-python3_7/sage_setup/docbuild/__init__.py:820: RemovedInSphinx30Warning: BuildEnvironment.frompickle() is deprecated. Please use pickle.load() instead.
  env = BuildEnvironment.frompickle(env_pickle, FakeApp(self.dir))
/usr/lib/python3.7/site-packages/sphinx/environment/__init__.py:740: RemovedInSphinx30Warning: BuildEnvironment.load() is deprecated. Please use pickle.load() instead.
  return cls.load(f, app)
[reference] Exception occurred:
[reference]   File "/usr/lib/python3.7/site-packages/sphinx/ext/intersphinx.py", line 207, in load_mappings
[reference]     for key, (name, (uri, invs)) in app.config.intersphinx_mapping.items():
[reference] ValueError: too many values to unpack (expected 2)

Doing some research indicates that this is a problem with the theme used in sphinx. Looking at the code, there are two themes used in Sage documentation, sage and sageref - the latter being a small variation of the former:

rg 'html_theme.*=' sage-9.0

sage-9.0/src/sage/docs/conf.py
215:html_theme = 'sage'
220:html_theme_options = {}
223:html_theme_path = [os.path.join(SAGE_DOC_SRC, 'common', 'themes')]
263:    html_theme_options['mathjax_macros'] = sage_mathjax_macros()

sage-9.0/src/doc/en/reference/conf_sub.py
54:html_theme = 'sageref'

In the output of rg (ripgrep, from sys-apps/ripgrep) above, you can see that a special theme for sphinx is set at two places, sage-9.0/src/sage/docs/conf.py and sage-9.0/src/doc/en/reference/conf_sub.py. The two themes, sage and sageref are under

sage-9.0/src/doc/common/themes/

According to sphinx-doc/sphinx#3075,

This is incompatible change since 1.4 (see #2320).
After the change, the entries was changed to a list of entryname, (links, subitems, key).

AFAIK, there were no migration path. So could you update your theme please?

That's where I stand now. I don't know how to implement the above advice into the two Sage themes - and I hope someone more knowledgeable in sphinx theming can give me a hand. In principle, that's all we would need to do to get rid of the

<dev-python/sphinx-2.0.0

constraint.

Come on, it's only the references that are left over! ✌️ :-)

[sedimentation-fault]

Since now I am not so sure whether the error is in the Sage 9.0 themes, or indeed in sphinx itself, I filed a bug report at the Sphinx project: sphinx-doc/sphinx#7409

[kiwifb]

Any help to migrate sage to sphinx-2+ is welcome. This is a major problem for a lot of people and any suggestion you have will be dutifully reported at https://trac.sagemath.org/ticket/28856

[sedimentation-fault]

There are news that the people at https://trac.sagemath.org/ticket/28856 should be informed of: the bug I opened at Sphinx (sphinx-doc/sphinx#7409) has resulted in a commit and a new suggestion from the developer:

It seems sagemath docs try to configure via Sphinx events. And it does not work since 2.x because we've refactored the intersphinx.
...
Please let them know to use config-inited event instead of builder-inited event. And I'll work to allow customizing intersphinx_mapping on the event on 3.0.

There are two notable things here:

1. Forget sphinx 2.x - it will not work for sage docs. Concentrate your efforts on v.3.
2. Use config-inited event instead of builder-inited event and things should eventually work out as intended, i.e. even if sage usage of sphinx is "different from normal", it should eventually work as is, because this kind of usage is now on the radar of the Sphinx developer for the 3.0 milestone.

Please report these great news in the thread you mentioned - progress is on the way! ✌️

[sedimentation-fault]

Following the above, from sphinx-doc/sphinx#7409:

I just merged #7415. It allows extensions to modify configurations via config-inited event. In this case, sagemath should be fixed to use the event and new Sphinx.

Please forward this to https://trac.sagemath.org/ticket/28856 - it looks very promising.

[kiwifb]

Yes I need to. I am spread between caring for my family in confinement (New Zealand), working from home and enabling people to work from home and rebuilding my gentoo home workstation [OS disk failed last evening. data is on a raid array :)]. I am experiencing building Gentoo from scratch for the first time in years.

[sedimentation-fault]

How on earth did you manage to get a disk to fail? I have never had a disk failing to me in the past 25 years or so - the last one was a 500MB disk that refused to spin and I had to give it a very focused lateral kick to get off again...

But I never used RAID extensively - although I do have one somewhere. You have to be careful with RAID - if the problem is not really the disk, but something deeper, you may risk destroying your one good copy.

Gentoo from scratch

Aah...the luxury of starting with an empty portage tree - never used it it the past 15 years...Lucky you - you certainly have less slot conflicts than I do right now... :-)

By all this I mean: take it easy, take your time. One thing at a time. First things first. There are worse problems - both in Gentoo and in real life.

[kiwifb]

T'was a SSD from 2011. I am somewhat surprised it lasted that long. I had the OS on the SSD and /home on the RAID - RAID 1 so the disks are mirrored and if one fail I can replace it without data loss. With one of my stash from an old decommissioned enterprise storage array.

I have managed a RAID6, ~250TB, storage array setup in 2011. I pulled the plug on the last controller in early February. I have replaced more disks than I care to mention [someone, somewhere, may have a count]

[kiwifb]

sphinx 3 will be used for sage 9.1 in sage-on-gentoo. It will be available in sage-9999 on the main branch soon if you absolutely need to move to a new sphinx before that release. sage-9999 on the master branch currently gives you sage 9.1.rc0.

I'll just remove sage-9.0 completely soon after sage 9.1 is released.

[kiwifb]

Just merged sphinx 3 support in sage-9999 on the master branch.

[kiwifb]

All good now.