idna 3.0 version package conflict

[naorlivne]

idna released version 3.0 but requests has a dependency on idna<3, this makes it impossible to keep up to date on both packages.

Expected Result

I want to be able to install the latest idna package alongside the latest requests package

Actual Result

ERROR: Cannot install -r requirements.txt (line 12) and idna==3.0 because these package versions have conflicting dependencies.

The conflict is caused by:

    The user requested idna==3.0

    requests 2.25.1 depends on idna<3 and >=2.5

Reproduction Steps

try to run pip install on a requirements.txt file with

requests==2.25.1
idna==3.0

System Information

multiple Python versions (3.6 up to 3.9) running on Docker containers inside Drone CI/CD

[jiasli]

As urllib3[secure] defines

https://github.com/urllib3/urllib3/blob/3f21165969b838fda29898cbd7218ac9578e319b/setup.py#L119

"idna>=2.0.0",

This causes pip 20.2.1 to show warning:

> pip install urllib3[secure] requests --force-reinstall
...
Collecting idna>=2.0.0; extra == "secure"
  Using cached idna-3.1-py3-none-any.whl (58 kB)
...
requests 2.25.1 requires idna<3,>=2.5, but you'll have idna 3.1 which is incompatible.

[jace]

This conflict between urllib3 and requests has left my automated builds broken for the last few weeks. We've been coping with manual deployments, but it hurts to not be able to use Travis CI. What's a recommended workaround?

[sethmlarson]

@jace urllib3[secure] requires idna>=2, requests requires idna>=2,<3, so if you install idna==2.10 there won't be any problem? Nothing should be breaking as a result of this, dependencies should be getting resolved by pip properly since version constraints are being set.

[ghost]

@sethmlarson
We should ignore this ERROR message from pip?

$ python3 -m pip install --upgrade idna      
Requirement already satisfied: idna in /usr/local/lib/python3.9/site-packages (2.5)
Collecting idna
  Using cached idna-3.1-py3-none-any.whl (58 kB)
Installing collected packages: idna
  Attempting uninstall: idna
    Found existing installation: idna 2.5
    Uninstalling idna-2.5:
      Successfully uninstalled idna-2.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
requests 2.25.1 requires idna<3,>=2.5, but you have idna 3.1 which is incompatible.
Successfully installed idna-3.1

[iambibhas]

@sethmlarson how long do we have to keep using older version of idna? This is essentially blocking us from using latest version of it. I don't think that should be suggested to anyone.

[RhinosF1]

Upgrading idna requires dropping python 2.7 so whenever that happens unless someone adds python_version markers to the dependancies

[naorlivne]

@RhinosF1 There's already a pull request with version markers waiting for approval/merging #5711 - my hope is to have it merged into the next release but as the conversation about it been locked for contributors only (which being my first PR for requests I'm not yet are) I can't make sure that will happen.

[RhinosF1]

@RhinosF1 There's already a pull request with version markers waiting for approval/merging #5711 - my hope is to have it merged into the next release but as the conversation about it been locked for contributors only (which being my first PR for requests I'm not yet are) I can't make sure that will happen.

Didn't see that, thanks!

[iambibhas]

The PR that fixes this and lets people use updated version of both packages has been blocked because this one entry might become a "maintenance nightmare" until requests drops Py2 support? The alternative being all the requests users will have a broken build or have to use a downgraded version of idna until requests decides to let us update both. That's just great.

[naorlivne]

@iambibhas That's not how I understood it, my understanding was that they are waiting for the next release for it and locked the conversation because they had too many people asking when it's due but I could be wrong

[iambibhas]

@naorlivne apologies. You're right. I missed parts of the thread. Will keep an eye for the release.

[almssp]

pip install idna==2.10
Looking in indexes: https://pypi.python.org/pypi/
ERROR: Could not find a version that satisfies the requirement idna==2.10
ERROR: No matching distribution found for idna==2.10

can't really install 2.10

[ViktorHaag]

Yes, it appears that the idna project has removed pre 3.x versions of the package from PyPi as a measure to remove support for Python 2 I believe -- they appear to have noticed that this has affected requests users and are tracking that with an issue on their side.

[sethmlarson]

@almssp @ViktorHaag The 2.x versions are still present and not yanked.

[ViktorHaag]

My apologies; I was looking in the wrong place... I looked at only the download files for the headrev release, and not the page for the older 2.10 release.

[Sebsebzen]

Version 2.05 works for me, but it's a bit annoying to get that error during build on Heroku. Any idea how I can ignore it?

[hartwork]

This has been blocking my update to latest idna for three months now (hartwork/wnpp.debian.net#13 (edit, and now hartwork/wnpp.debian.net#116)).
If someone at @psf wants to team up on fixing this, please contact me through e-mail, maybe I can help.

[naorlivne]

@hartwork #5711 will fix this, @psf are just sitting on it until the next release

[jace]

🙏 Is there an ETA for when #5711's patch goes into a PyPI release?

[nateprewitt]

@jace We do not currently have a solidified date but are working on getting things ready. You can follow the status of #5868 for when we perform the release.