explore python dependency management

[alangenfeld]

Problem: dependencies are encoded explicitly and implicitly in several places - setup.py, requirements.txt, tox.ini, dev env setup and CI jobs. This sucks.

We should have a single source of truth and then use that to generate what we need. In addition to solving our current problems, a good system here can empower stuff like determining what modules (and their recursive deps) to test as a function of changed files.

[alangenfeld]

https://github.com/bazelbuild/rules_python

[alangenfeld]

at minimum should we move to something with a lockfile like pipenv?

[mgasner]

https://github.com/sdispater/poetry

[alangenfeld]

relevant incident report:

https://scikit-learn.org/stable/whats_new.html#version-0-21-0 released which broke all of our builds since it dropped support for anything Python 3.4 and below https://github.com/scikit-learn/scikit-learn/releases

We had some slowness fixing this due to not getting the order of editable installs in one place, see https://dagster.phacility.com/D75

[mgasner]

#1567

[mgasner]

python-poetry/poetry#34

[mgasner]

https://poetry.eustace.io/docs/versions/#path-dependencies

[mgasner]

https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/

[mgasner]

python-poetry/poetry#1123

[mgasner]

We could check in requirements.txt files created by pip freeze, etc., but I think this sends the wrong message (you must develop using these specific versions) for an open source project.

[mgasner]

https://hynek.me/articles/python-app-deps-2018/

[mgasner]

tox:

whitelist_externals = poetry
skip_install = true
commands =
    poetry install -v
    poetry run pytest tests/

https://medium.com/@DJetelina/making-a-python-library-how-the-ecosystem-changed-in-2-5-years-a3b5eb16d7df

[mgasner]

RTD issues:
readthedocs/readthedocs.org#4912

[mgasner]

python-poetry/poetry#1245 is now fixed on poetry:develop

[mgasner]

cd ../dagster && rm poetry.lock && poetry install
cd ../dagster-graphql && rm poetry.lock && poetry install
cd ../dagit && rm poetry.lock && poetry install

[mgasner]

https://dephell.org/

[mgasner]

https://frostming.com/2019/01-04/pipenv-poetry

[orsinium]

I recommend to have a look at the next combination:

• poetry as format
• dephell to generate requirements.txt/setup.py for compatibility
• dephell to run tests (on CI) too.

Related docs:

• https://dephell.readthedocs.io/cmd-deps-convert.html
• https://dephell.readthedocs.io/cmd-generate-travis.html

Dephell's own config can be helpful: https://github.com/dephell/dephell/blob/master/pyproject.toml

The main idea is to make as many environments as possible with dephell environments and extra dependecies. One environment for one command. For example, test environment for dephell:

[tool.dephell.pytest]
from = {format = "poetry", path = "pyproject.toml"}
envs = ["main", "tests"]
command = "python -m pytest tests/"

...

[tool.poetry.dev-dependencies]
pytest = "*"
requests-mock = "*"
aioresponses = "*"
...

[tool.poetry.extras]
tests = ["aioresponses", "pytest", "requests-mock"]

[schrockn]

Wow @orsinium dephell might be the tool I've been waiting for. All of our setup.py's and requirements.txt's under code-generation is definitely what I think the structure should be. Thanks a ton for the pointer.

[natekupp]

Per discussion here: https://dagster.phacility.com/D1750 we should also update .buildkite/images/docker/snapshot-update.sh as part of our dependency management refresh / this project.

[mgasner]

python-poetry/poetry#1245 has landed (in the new poetry-core package) and dev on poetry looks like it has picked up again -- i'll revisit this (with an eye to moving to poetry/dephell) soon

[mgasner]

current dependency hell:

ERROR: apache-airflow 1.10.9 has requirement future<0.17,>=0.16.0, but you'll have future 0.18.2 which is incompatible.
ERROR: apache-airflow 1.10.9 has requirement pandas<1.0.0,>=0.17.1, but you'll have pandas 1.0.3 which is incompatible.
ERROR: botocore 1.15.32 has requirement docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.
ERROR: boto3 1.9.253 has requirement botocore<1.13.0,>=1.12.253, but you'll have botocore 1.15.32 which is incompatible.
ERROR: snowflake-connector-python 2.2.2 has requirement botocore<1.15,>=1.5.0, but you'll have botocore 1.15.32 which is incompatible.

[shao-shuai]

current dependency hell:

ERROR: apache-airflow 1.10.9 has requirement future<0.17,>=0.16.0, but you'll have future 0.18.2 which is incompatible.
ERROR: apache-airflow 1.10.9 has requirement pandas<1.0.0,>=0.17.1, but you'll have pandas 1.0.3 which is incompatible.
ERROR: botocore 1.15.32 has requirement docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.
ERROR: boto3 1.9.253 has requirement botocore<1.13.0,>=1.12.253, but you'll have botocore 1.15.32 which is incompatible.
ERROR: snowflake-connector-python 2.2.2 has requirement botocore<1.15,>=1.5.0, but you'll have botocore 1.15.32 which is incompatible.

I got the pandas error as well, did you find solution?

[mgasner]

I think we should do a couple of things in concert over the Xmas holidays when hopefully the rate of commits slows down a little: move to poetry with pants to manage monorepo builds (deprecating some of our homegrown build scripting), use cookiecutter or copier for project templating. This should also set us up for a cleaner conda story.

some refs
https://www.pantsbuild.org/docs/how-does-pants-work
https://github.com/mitodl/ol-data-pipelines/blob/putting_on_pants/pants.toml
https://cookiecutter.readthedocs.io/en/1.7.2/
https://copier.readthedocs.io/en/stable/
https://github.com/dojeda/poetry2conda

[natekupp]

@mgasner definitely onboard w/ poetry.

thoughts on pants vs. bazel vs. please.build? maybe worth a comparison/eval?

[mgasner]

tried to use poetry in the internal stuff and its editable installs still don't work right

[orsinium]

I started to use flit for my projects.

• Nice format
• Simple tool with only a few essential subcommand
• A few ways to do the editable installation (symlinking or good old pth)
• PEP-517 makes it pip-compatible.
• PEP-631 was accepted and flit will definitely support it
• No lockfiles but pip-tools will support PEP-517 in a few days: Use pep517 to load project metadata jazzband/pip-tools#1311

To automate things like venv creation and deps installation, I use taskfile.
Example: https://github.com/life4/deal/blob/master/Taskfile.yml
Installation on CI is also easy: https://github.com/life4/deal/blob/master/.drone.star#L13-L19

[mgasner]

fwiw poetry is much improved from when i looked at it in december 2020, editable installs now seem to work

[sryza]

We've made some big changes since this was filed and it's mandate is very general. Closing.