Unpin docs dependencies

[edsu]

The sphinx doc build run by CircleCI is currently failing:

make: Entering directory '/home/circleci/wayback/docs'
Traceback (most recent call last):
  File "/home/circleci/venv/bin/sphinx-build", line 5, in <module>
    from sphinx.cmd.build import main
  File "/home/circleci/venv/lib/python3.8/site-packages/sphinx/cmd/build.py", line 25, in <module>
    from sphinx.application import Sphinx
  File "/home/circleci/venv/lib/python3.8/site-packages/sphinx/application.py", line 42, in <module>
    from sphinx.registry import SphinxComponentRegistry
  File "/home/circleci/venv/lib/python3.8/site-packages/sphinx/registry.py", line 24, in <module>
    from sphinx.builders import Builder
  File "/home/circleci/venv/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 26, in <module>
    from sphinx.util import import_object, logging, rst, progress_message, status_iterator
  File "/home/circleci/venv/lib/python3.8/site-packages/sphinx/util/rst.py", line 22, in <module>
    from jinja2 import environmentfilter
ImportError: cannot import name 'environmentfilter' from 'jinja2' (/home/circleci/venv/lib/python3.8/site-packages/jinja2/__init__.py)
make: *** [Makefile:20: html] Error 1
make: Leaving directory '/home/circleci/wayback/docs'

Exited with code exit status 2

CircleCI

It looks like this might be the result of jinja2 imports being deprecated?

sphinx-doc/sphinx#10291

I tried unpinning all the dependencies, and making a few minor changes to the sphinx config and the docs seem to build fine locally but maybe I'm missing some detail about theme/css?

Unfortunately I did have to tell pip to ignore sphinx 5.2.0.post0 which appears to have a recent bug that hopefully will be fixed shortly:

readthedocs/sphinx_rtd_theme#1343

[edsu]

Hmm, I'm confused why the docs tasks is installing an old version of Sphinx. When I run these pip install steps locally I get Sphinx v.5.2.0, but the CircleCI build gets v3.1.2?

https://app.circleci.com/pipelines/github/edgi-govdata-archiving/wayback/267/workflows/fec11447-9a0b-4fa5-8c17-d8327536f00e/jobs/876?invite=true#step-105-74

[edsu]

Removing the CircleCI caching of the venv fixed the stale dependencies problem. Again, I'm not sure what the rationale was for doing that, so please feel free to close this PR without merging.

[Mr0grog]

I’m not sure unpinning things is really the right solution here, but should have some time tomorrow to take a closer look and figure this out.

Removing the CircleCI caching of the venv fixed the stale dependencies problem. Again, I'm not sure what the rationale was for doing that

We should not remove the cache. This is basic good citizen behavior — the growth of CI services like Circle, Travis, GitHub Actions, etc. is a good thing, but people not caching imposed out-of-control cost growth on PyPI.

I think the cache was also not your problem here. Pip still runs and will install upgrade anything where the requirement file disagrees with what’s already on disk from the cache (and the cache key includes a hash of the two requirements files, so it will change and sort of auto-upgrade when you edit those files). The last build that had Sphinx 3 did so because you only specified !=5.2.0.post0, which the cache would have satisfied by including v3, and pip would have seen no reason to upgrade it based on the constraints you put in.

Re: Sphinx 3 vs. 5, I actually don’t think I’ve upgraded any packages I maintain to Sphinx 5, because it has a conflict with either Pyflakes or Flake8 (I’ve forgotten which) that makes for kind of a mess. That said, I’d thought I had at least upgraded this to Sphinx 4.x. ¯\_(ツ)_/¯

[Mr0grog]

Closing this in favor of #91 (it’s a bigger change, but setting the right dependency versions is critical if we don’t want various features in our docs, lint rules, etc. to randomly and maybe silently break).