Upgrade Python requirements

[andersk]

Upgraded:

• arrow: 0.17.0 → 1.0.3
• attrs: 20.3.0 → 21.2.0
• babel: 2.9.0 → 2.9.1
• black: 20.8b1 → 21.5b0
• boto3: 1.17.53 → 1.17.69
• botocore: 1.20.53 → 1.20.69
• cachetools: 4.2.1 → 4.2.2
• dataclasses-json: 0.5.2 → 0.5.3
• django-formtools: 2.2 → 2.3
• django-otp: 1.0.3 → 1.0.4
• django[argon2]: 3.2 → 3.2.2
• gitlint: 0.15.0 → 0.15.1
• moto[s3]: 2.0.5 → 2.0.6
• openapi-core: 0.13.7 → 0.13.8
• pep517: (new) → 0.10.0
• phonenumberslite: 8.12.21 → 8.12.22
• pip: 20.2.4 → 20.3.4
• pip-tools: 5.5.0 → 6.1.0
• pygments: 2.8.1 → 2.9.0
• pyre-check: 0.9.0 → 0.9.1
• queuelib: 1.5.0 → 1.6.1
• responses: 0.13.2 → 0.13.3
• s3transfer: 0.3.7 → 0.4.2
• semgrep: 0.47.0 → 0.50.1
• sentry-sdk: 1.0.0 → 1.1.0
• setuptools: 56.0.0 → 56.1.0
• six: 1.15.0 → 1.16.0
• tlds: 2021041302 → 2021050600
• twilio: 6.56.0 → 6.58.0
• typing-extensions: 3.7.4.3 → 3.10.0.0
• xmlsec: 1.3.9 → 1.3.10
• zulip: zulip/python-zulip-api@175972c → zulip/python-zulip-api@4d482e0
• zulip-bots: zulip/python-zulip-api@175972c → zulip/python-zulip-api@4d482e0

I also decided to audit the requirements that are being held back:

• arrow: 1.0.3 ↛ 1.1.0
  • pinned by gitlint: Unpin requirements.txt jorisroovers/gitlint#162
• defusedxml: 0.6.0 ↛ 0.7.1
  • pinned by python3-saml: Remove the dependency on defusedxml SAML-Toolkits/python3-saml#236, fixed in Git
• django-auth-ldap: 2.0.0 ↛ 2.4.0
  • forked for Add feature to find users in the directory by email django-auth-ldap/django-auth-ldap#150
• docutils: 0.16 ↛ 0.17.1
  • bounded by Sphinx: Support docutils-0.17 sphinx-doc/sphinx#9125, fixed in beta release
  • bounded by sphinx_rtd_theme: broken alignment for captions in toctrees in ver. 0.5.1 readthedocs/sphinx_rtd_theme#1112
• h2: 2.6.2 ↛ 4.0.0
  • bounded by hyper, which is unmaintained but required by apns2: hyper dependency is no longer maintained Pr0Ger/PyAPNs2#126
• hpack: 3.0.0 ↛ 4.0.0
  • bounded by h2
• hyperframe: 3.2.0 ↛ 6.0.1
  • bounded by h2 and hyper
• idna: 2.10 ↛ 3.1
  • bounded by moto: Add different versions of idna depending on the python version getmoto/moto#3908
  • bounded by requests: idna 3.0 version package conflict psf/requests#5710
• ipython: 7.16.1 ↛ 7.23.1
  • needs Python 3.7
• jedi: 0.17.2 ↛ 0.18.0
  • incompatible with ipython < 7.20: IPython (<=7.19) incompatible with jedi 0.18.0 davidhalter/jedi#1714, Last jedi release (0.18.0) is incompatible with ipython (7.19 and 7.18 tested); reason - column arg was deprecated, and now removed ipython/ipython#12740
• line-profiler: 3.1.0 ↛ 3.2.6
  • ipython version bound typo: Wrong IPython version in requirements/runtime.txt for Python 3.6 pyutils/line_profiler#77
• openapi-core: 0.13.8 ↛ 0.14.0
  • KeyError: 'type' regression: [0.14.0 regression] param['schema']['type'] in ['array', 'object'] raises KeyError: 'type' python-openapi/openapi-core#322
• parso: 0.7.1 ↛ 0.8.2
  • bounded by jedi, fixed in jedi 0.18.0
• pip: 20.3.4 ↛ 21.1.1
  • breaks our hack for installing specific commits from Git: Updating remote links with new URLs for PEP508 functionallity pypa/pip#5780
• PyJWT: 1.7.1 ↛ 2.1.0
  • bounded by apns2: apns2.credentials: since version 2.0.0 pyJWT encode method return str Pr0Ger/PyAPNs2#122, fixed in Git
  • pinned by twilio: Upgrade to PyJWT 2.0 twilio/twilio-python#556
• Scrapy: 2.4.1 ↛ 2.5.0
  • needs h2 3.0
• social-auth-core: 4.0.2 ↛ 4.1.0
  • needs PyJWT 2
• SQLAlchemy: 1.3.24 ↛ 1.4.14
  • badly busted type annotations
• tornado: 4.5.3 ↛ 6.1
  • we seem to have decided an obscure logging feature is more important than upgrading a three-year-old library?: Upgrade tornado to latest version #8913
• traitlets: 4.3.3 ↛ 5.0.5
  • needs Python 3.7

[timabbott]

lgtm. Thanks for doing that audit! I think for Tornado, we should figure out an upgrade plan soon; I don't think we need the specific logging feature we have, but we need some way to know whether a Tornado process is nearly overloaded.

And the other thing we should prioritize is removing PyAPNS2: it's clear that library is resulting in a tangle out outdated dependencies. That one is likely easier, in that we don't have a clear blocker for it and have never been happy with the library.

Is there an upstream issue for preserving our pip hack and/or some plan there?

[andersk]

I don't think we need the specific logging feature we have, but we need some way to know whether a Tornado process is nearly overloaded.

Is there some more end-to-end metric that might be more useful, like the amount of time spent waiting on a message to come in from RabbitMQ?

Is there an upstream issue for preserving our pip hack and/or some plan there?

There are a number of locked pip issues such as pypa/pip#2837, but the currently active one seems to be pypa/pip#5780, which I’ve added above.

I guess a partway upgrade to pip 20.3 with --use-deprecated=legacy-resolver is better than nothing, so I’ll do that.

[timabbott]

Is there some more end-to-end metric that might be more useful, like the amount of time spent waiting on a message to come in from RabbitMQ?

The goal is to have a leading indicator of the form "we currently have 0 queuing, but if we added 20% more load, we'd have queuing". Essentially any amount of queueing is super bad. I think anything that lets us estimate that will work; amount of time spent asleep/waiting would certainly work (but we don't wait just for RabbitMQ, since we get HTTP requests too).