Implement merge_domaindata() on Zeek domain to support parallel builds

[awelzel]

Only the resulting environment.pickle file appears different.

$ diff -ru ./single-proc-html/ ./build/html/
Binary files ./single-proc-html/.doctrees/environment.pickle and
./build/html/.doctrees/environment.pickle differ
$

I am not quite sure incremental updates will work right,
but this may be a start.

Changes build time locally from 8m 40s to 2m 50s.

Closes #122, maybe.

[awelzel]

@timwoj - this doesn't seem to have much of an effect on the github action timings. Do you see a difference in runtime locally?

[timwoj]

On macOS I get the following warning. It's maybe 5 seconds faster on your branch, but that's probably just that it had cached part of it in RAM on the second run.

For security reason, parallel mode is disabled on macOS and python3.8 and above. For more details, please read sphinx-doc/sphinx#6803

On Linux, it's wildly faster. master took 423 seconds and your branch took 122 seconds. That's with 8 CPUs though. If I force it down to 2 CPUs similar to how it is on GitHub, it takes 448 seconds.

[awelzel]

The OSX piece is sad. It seems to have been fixed with Sphinx 4.4 though:

sphinx-doc/sphinx#6881 (comment)

On Linux, it's wildly faster. master took 423 seconds and your branch took 122 seconds. That's with 8 CPUs though. If I force it down to 2 CPUs similar to how it is on GitHub, it takes 448 seconds.

Yep, it doesn't seem -j scales quite as nice as maybe expected.

I wonder if caching the build dir could be an alternative, but not sure how safe/reliable that works.

[timwoj]

I installed a newer version of sphinx into a virtualenv on macOS here and can confirm that it seems they fixed parallel builds on here now. This is what I have in requirements.txt now:

#jinja2<3.1
#pygments>=2.12.0
#docutils==0.16
sphinx_rtd_theme==1.0.0
Sphinx==5.0.2
GitPython==3.1.27

docutils didn't get installed at all. jinja2 3.1.2 and pygments 2.12.0 got automatically installed. It seems that the newer version of Sphinx also fixed whatever conflict they were having with the newer version of jinja2. I was also able to bump the sphinx_rtd_theme package up to the latest version.

[awelzel]

After writing some documentation and building it quite regularly and being annoyed about the lengthy process, I included the version updates and cleaned up the -j passing a bit in the Makefile. Otherwise no changes. I skimmed the produced docs and they seemed okay.

docutils didn't get installed at all

It installed for me as 0.17.1, so I kept that in the requirements.txt.

Should we merge this and see if something breaks? 🤷‍♂️

[timwoj]

Sounds good to me. I scrolled through the changes as well and it looks like mostly upgrades to how Sphinx generates both the CSS and the JS that drive the site. I don't think there's anything significant with regards to the docs themselves here.

I was curious if RTD had any problem with the new python modules, but I doubt it will. I triggered a build off it just in case: https://docs.zeek.org/en/topic-awelzel-parallel-doc-build/. The only thing I can't figure out is how to tell RTD to pass the -j flag to the build. I don't think it did since the master build I did right before it only took about 8 seconds. The command it runs to start the build is below:

 /home/docs/checkouts/readthedocs.org/user_builds/zeek-docs/envs/topic-awelzel-parallel-doc-build/bin/python -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html 

[timwoj]

We can look at the RTD thing separately.