pickling issues building sage-doc-9999 with use=doc-pdf and all available LANGS

[strogdon]

The most pressing situation seems to be on Prefix although I occasionally have issues on Gentoo. The pickling issues are similar to #667 (comment) when building the standard sage-doc-9999. Doing

emake doc-html PYTHON=${PYTHON}

seems to work and

emake doc-pdf PYTHON=${PYTHON}

also appears to work. This latter case appears to build some html stuff under html/en. The following

emake doc-html PYTHON=${PYTHON} && emake doc-pdf PYTHON=${PYTHON}

fails with

MAKE=make /storage/strogdon/gentoo-rap/usr/bin/python3.10 -m sage_docbuild reference/spkg pdf --no-prune-empty-dirs
[tensor_fr] dumping search index in English (code: en)... done
[tensor_fr] The HTML pages are in ../../../build_doc/html/en/reference/tensor_free_modules.
Build finished. The built documents can be found in /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-doc-9999/work/build_doc/html/en/reference/tensor_free_modules
MAKE=make /storage/strogdon/gentoo-rap/usr/bin/python3.10 -m sage_docbuild reference/manifolds pdf --no-prune-empty-dirs
Error building the documentation.
Traceback (most recent call last):
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__main__.py", line 2, in <module>
    main()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 1745, in main
    builder()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 247, in pdf
    self.latex()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 776, in _wrapper
    for module_name in self.get_new_and_updated_modules():
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 946, in get_new_and_updated_modules
    env = self.get_sphinx_environment()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.10/site-packages/sage_docbuild/__init__.py", line 843, in get_sphinx_environment
    env = pickle.load(f)
_pickle.UnpicklingError: pickle data was truncated

    Note: incremental documentation builds sometimes cause spurious
    error messages. To be certain that these are real errors, run
    "make doc-clean doc-uninstall" first and try again.
[repl     ] dumping search index in English (code: en)... done
[repl     ] The HTML pages are in ../../../build_doc/html/en/reference/repl.
Build finished. The built documents can be found in /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-doc-9999/work/build_doc/html/en/reference/repl

The point where the failure occurs is not predictable. So far I have been unable to build html + pdf docs for 9.6.beta3 on Prefix.

[kiwifb]

I had to build at -j6 instead of -j8 with just two languages. But I squarely got a #667 (comment) kind of issue. The case that give errors would be the standard vanilla procedure. Default make produces html doc and then you have to do make doc-pdf to get the pdf, although there is probably a way to request it from the top.

I am assuming that what I put in which should be make doc-html doc-pdf doesn't work either?

[strogdon]

Yes, when I do the make doc-html doc-pdf I get the #667 (comment) kind of issue. Not sure why the make doc-html and then make doc-pdf fails. I googled the pickle error above and it may appear when a pickle is empty.

[kiwifb]

It is just another kind of parallel make issue where a task starts before the necessary previous one has ended. But where to tweak?

[strogdon]

It appears that when one does make doc-html and then make doc-pdf that some html stuff (html/en) is rebuilt. I don't think this should be the case. Shouldn't the make process know the html/en is up to date and thus not rebuild it. Maybe they are different make processes. FWIW I do have an 8-core machine with MAKEOPTS="-j9" where sage-doc-9999 built just fine.

[kiwifb]

Indeed some bits of html are rebuilt and in an ideal world they shouldn't. make does look for files as targets, when they are more abstract like it is the case here you can have those kind of issues. In fact in an ideal world running make doc-html would just check a number of files are present and do nothing. This is not the case at present. If we had a clear file as a final target we could avoid that kind of re-build.

[strogdon]

The host of the prefix has 8 cores. I'm only able to get sage-doc-9999 to build serially, -j1. It's very tedious.

[kiwifb]

Sounds seriously tough. I really need to bring back pre-build doc in stable if we cannot fix or mitigate that.

[kiwifb]

Latest merge on the vbraun branch from 6 hours ago, doesn't have anything about sage_docbuild or sage-doc, and it just build the doc like there was no problems. Unlike the previous merge which would have been identical to 9.6.beta3.

[strogdon]

Some thoughts. When USE=doc-pdf, in src/doc/Makefile remove respectively, doc-inventory-reference and doc-html-reference from the targets doc-pdf-reference and doc-pdf-other and build the docs as

emake doc-html PYTHON=${PYTHON} && emake doc-pdf PYTHON=${PYTHON}

then each component html and pdf builds in parallel - html first and pdf next. Not strictly in parallel but it does build on the Prefix.

[kiwifb]

Got hit on the latest vbraun merge. '-j8' unknown character, '-j6' 'FileNotFoundError'. So, I am going to try what you are suggesting in your latest message to see if things improve.

[kiwifb]

571d34b helped. At a cost of 2 minutes of build time (15mn instead of 13) probably a bargain. It is not exactly pleasant in my opinion but it may have to do for now.

[strogdon]

One machine where sage-doc-9999 from the overlay built. Two machines, one the subject Prefix, needed the fix to build.