upgrade python

[jspaaks]

Refs #474

Upgraded all pythons from 3.6 to 3.9 but doesnt work:

$ docker-compose exec harvesting python app.py harvest all
Traceback (most recent call last):
  File "/app/app.py", line 8, in <module>
    from zotero import get_mentions
  File "/app/zotero.py", line 10, in <module>
    from pyzotero import zotero
  File "/usr/local/lib/python3.9/site-packages/pyzotero/zotero.py", line 43, in <module>
    import feedparser
  File "/usr/local/lib/python3.9/site-packages/feedparser.py", line 93, in <module>
    _base64decode = getattr(base64, 'decodebytes', base64.decodestring)
AttributeError: module 'base64' has no attribute 'decodestring'

Trying Python 3.8 next.

$ docker-compose build && docker-compose up -d && docker-compose logs --follow

shows

rsd-backend      | [2020-10-14 17:16:31 +0000] [1] [INFO] Starting gunicorn 19.7.1
rsd-backend      | [2020-10-14 17:16:31 +0000] [1] [INFO] Listening at: http://0.0.0.0:5001 (1)
rsd-backend      | [2020-10-14 17:16:31 +0000] [1] [INFO] Using worker: sync
rsd-backend      | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-backend      |   return io.open(fd, *args, **kwargs)
rsd-backend      | [2020-10-14 17:16:31 +0000] [9] [INFO] Booting worker with pid: 9
rsd-backend      | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-backend      |   return io.open(fd, *args, **kwargs)
rsd-backend      | [2020-10-14 17:16:31 +0000] [10] [INFO] Booting worker with pid: 10
rsd-backend      | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-backend      |   return io.open(fd, *args, **kwargs)
rsd-backend      | [2020-10-14 17:16:31 +0000] [11] [INFO] Booting worker with pid: 11

and

rsd-frontend     | /src/app/application.py:100: SyntaxWarning: "is not" with a literal. Did you mean "!="?
rsd-frontend     |   if field in software and software[field] and software[field] is not '':
rsd-frontend     | [2020-10-14 17:16:34 +0000] [1] [INFO] Starting gunicorn 19.7.1
rsd-frontend     | [2020-10-14 17:16:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:5004 (1)
rsd-frontend     | [2020-10-14 17:16:34 +0000] [1] [INFO] Using worker: sync
rsd-frontend     | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-frontend     |   return io.open(fd, *args, **kwargs)
rsd-frontend     | [2020-10-14 17:16:34 +0000] [9] [INFO] Booting worker with pid: 9
rsd-frontend     | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-frontend     |   return io.open(fd, *args, **kwargs)
rsd-frontend     | [2020-10-14 17:16:34 +0000] [10] [INFO] Booting worker with pid: 10
rsd-frontend     | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-frontend     |   return io.open(fd, *args, **kwargs)
rsd-frontend     | [2020-10-14 17:16:34 +0000] [11] [INFO] Booting worker with pid: 11

and

rsd-authentication | [2020-10-14 17:16:28 +0000] [1] [INFO] Starting gunicorn 19.7.1
rsd-authentication | [2020-10-14 17:16:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:5002 (1)
rsd-authentication | [2020-10-14 17:16:28 +0000] [1] [INFO] Using worker: sync
rsd-authentication | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-authentication |   return io.open(fd, *args, **kwargs)
rsd-authentication | [2020-10-14 17:16:28 +0000] [7] [INFO] Booting worker with pid: 7
rsd-authentication | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-authentication |   return io.open(fd, *args, **kwargs)
rsd-authentication | [2020-10-14 17:16:28 +0000] [8] [INFO] Booting worker with pid: 8
rsd-authentication | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
rsd-authentication |   return io.open(fd, *args, **kwargs)
rsd-authentication | [2020-10-14 17:16:28 +0000] [9] [INFO] Booting worker with pid: 9

So it looks like frontend runs into this error: benoitc/gunicorn#2091 although we have gunicorn==19.7.1, in frontend, backend and authentication. Besides the warnings in the log, it seems to work fine though, at least all the parts from "Verifying the local installation" work as expected.

The syntax error about is not seems a real error, I've changed it as suggested.

Based on gunicorns changelog, we should be able to update to 20.x. The breaking changes don't seem relevant to us. https://docs.gunicorn.org/en/latest/news.html

After updating gunicorn: Logs look clean now, also when I do a harvest all. "Verifying the local installation" works as expected. but I saw #518 in the logs. From a user perspective, everything works though.

[jspaaks]

Just update Python they said, it'll be easy they said. Well it was a nice 🐰 🕳️.

[fdiblen]

I had no issues with a clean build of docker images.
Good work.
Tssss... 🐍

[jspaaks]

Thank you 🥧