fix some CI issues (#636)

[willkg]

Fix doctest linting issues with migrating chapter (#636)

This renames the file, adds it to the index, and reworks the doctest blocks which won't work in our docs build environment so they're code blocks instead. That does mean they're untested, but that's much easier to support for now.

Fix pip-tools problem and rework requirements (#636)

This fixes the problem with pip-tools < 6.5.0 and pip 22 by adding pip-tools to the dev requirements, setting it to 6.5.0, and installing that before checking the requirements.

This also moves requirements checking code to the scripts/run_tests.sh script so it's easier to do in a local dev environment.

This also rebuilds the dev requirements.

[willkg]

The 3.6 tests (3) are failing because we've got dependencies that don't work with that. I'm pretty sure #629 will fix that, so I'm not going to bother with those.

The 3.7 tests (3) are failing with this:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    typed-ast>=1.4.2 from https://files.pythonhosted.org/packages/05/3f/c224ee002bd0f5a763c54004198c321b30cdd67b319a1d4d69b26fdc19c2/typed_ast-1.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl#sha256=f30ddd110634c2d7534b2d4e0e22967e88366b0d356b24de87419cc4410c41b7 (from black==22.1.0->-r requirements-dev.txt (line 19))

I think we can fix this by building requirements-dev.txt using Python 3.7 rather than Python 3.9.

The Windows checks (3) fail with this:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pywin32-ctypes!=0.1.0,!=0.1.1 from https://files.pythonhosted.org/packages/9e/4b/3ab2720f1fa4b4bc924ef1932b842edf10007e4547ea8157b0b9fc78599a/pywin32_ctypes-0.2.0-py2.py3-none-any.whl#sha256=9dc2d991b3479cc2df15930958b674a48a227d5361d413827a4cfd0b5876fc98 (from keyring==23.5.0->-r requirements-dev.txt (line 193))

I'm not sure offhand how to fix that. Maybe a platform-specific requirements-dev.txt file? Ugh.

The pypy checks (3) fail with the same issue as the 3.6 tests:

ERROR: Could not find a version that satisfies the requirement filelock==3.4.2 (from versions: 0.2.0, 0.2.1, 0.2.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 2.0.0, 2.0.1, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.0.8, 2.0.9, 2.0.10, 2.0.11, 2.0.12, 2.0.13, 3.0.0, 3.0.2, 3.0.3, 3.0.4, 3.0.6, 3.0.8, 3.0.9, 3.0.10, 3.0.12, 3.1.0, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.3.2, 3.4.0, 3.4.1)
ERROR: No matching distribution found for filelock==3.4.2

Stepping back a bit, I'm not sure we should be doing a requirements-dev.txt at all and instead should put the top-level requirements in setup.py. That's what I'm doing with all my other projects. It's generally easier to maintain. Issue #620 covers that.

I'll think more about this tomorrow and talk with @g-k about issue #620.

[g-k]

+1 on dropping 3.6 it'll make cleaner type syntax available to use too.

building requirements-dev.txt using Python 3.7 rather than Python 3.9

wfm

Maybe a platform-specific requirements-dev.txt file? Ugh.

Yeah, kinda gross. I've had to do this for other projects.

Commented 👍 on #620

[willkg]

This fixes a couple of issues. The rest of the issues will get fixed by work on issue #620 and landing PR #629.

@g-k Can you review this?

[willkg]

I moved pip-tools to be a requirement because it's easier/better to maintain all the dev requirements in one place and that's a dev requirement. Thus we need to install the requirements before comparing pip-compile output.

[willkg]

This chapter wasn't included in the index before, but it should have been.

[willkg]

In order for these to be doctest blocks and pass our tests, we'd have to install html5lib as a dev requirement and I don't want to do that because I think it'll make it harder to maintain everything if html5lib is both a dev requirement and also a vendored package.

I don't want to use the vendored version because it makes these illustrative examples harder to understand.

Further, these examples are missing bits, so we'd have to flesh them out as complete examples.

Given all that, I thought it'd be easier to just turn them into code blocks for now.

[willkg]

pip-tools 6.5.0 fixes the issue that prevents it from working with pip 22, so we should at least require 6.5.0.

[willkg]

This is curious. The github workflow had comparing running with all the python versions, but pip-compile puts the Python version number in the file, so it'd never diff without raising issues.

I ended up rebuilding the file with Python 3.9, but maybe we should do it with 3.7 and some of the issues go away? I don't know. I think the check for determining whether the requirements-dev.txt file is updated isn't working right. Maybe it should be in the lint workflow instead since that only runs with one version of Python?

[g-k]

+1 on using 3.7 or moving dev deps to a dev pkg

[willkg]

I added this because it's difficult to have non-trivial bits in the CI workflows that we can't easily run in a local dev environment. However, I'm not thrilled with how this works. pip-compile is a little finicky about things.

Maybe it makes more sense to copy things to a tmp directory, compile it there, and then diff it?

[g-k]

r+

[g-k]

+1 on using 3.7 or moving dev deps to a dev pkg

[willkg]

Thank you!