Drop Python 2 support

[haikuginger]

I'm wondering whether we should start a conversation with our biggest partners (pip, Requests, et. al.) around what the long-term plan for Python 2 ought to be.

The platform itself is planned to have no further support after 2020, so we should probably have a transition plan laid out well in advance so that we can start providing notification and documentation to our users.

Obviously, what that plan ought to be, or even whether we should start considering it yet, is up for discussion.

[sigmavirus24]

Hey @haikuginger

So I'm 100% on board with killing off Python 2 today, but we can't. We will likely have to continue supporting it until 2020 or past that date. urllib3 and requests are both (what is frequently called) "core infrastructure". This is because of pip's dependency on both. What this means to me is that until pip can stop supporting Python 2, we can not drop that support.

I don't think we can make a plan now, three years out, because there's three years of data to be collected around usage and that absolutely should factor into our decision making then.

If we want to make a plan to make a decision, that's fine by me. Deciding what our deprecation path now is imprudent. I don't think we'll be able to make a decision without that data.

[shazow]

Chatted with @sigmavirus24 and folks IRL today, and it seems like a reasonable ambition to drop support for a Python in urllib3 v2.0 (added to the wishlist) which will be a big backwards-compat breaking set of changes.

Not sure when that will happen, though. Hopefully sometime before 2020.

[haikuginger]

Sounds good - so we'll basically plan to release a 2.0 version with a brand-new codebase that drops Python 2.x support, but provide extended maintenance support for 1.x as long as pip needs it?

Just making sure I understand the situation; wish I could have made it out to PyCon, but there you are.

[shazow]

For some definition of extended maintenance, yes. :P

[haikuginger]

@sigmavirus24 @Lukasa, it looks like pip has dropped support for Python 2.6. How does the Requests team feel about doing so as well in coordination with urllib3?

[nateprewitt]

@haikuginger the current decision was to drop Python 2.6 with Requests 3.0 and we have a PR doing so merged into that branch. I think some of the changes slated for Requests 3.0 were dependent on urllib3 2.0 though.

I'm not sure what the decision is for the future of the urllib3 2.0 branch, but once that's figured out, we can probably start to roadmap out related Requests 3.0 changes. If urllib3 2.0 is still far off, and we have a substantial reason to drop 2.6, we could consider dropping it earlier. However, we'd probably want at least a full release cycle for deprecation before doing that.

P.S. Congrats on lead maintainer 🎉

[hugovk]

By the way, here's the pip installs for urllib3 from PyPI for April 2018:

python_version	percent	download_count
2.7	56.40%	3,326,214
3.6	27.50%	1,621,613
3.5	9.88%	582,880
3.4	5.33%	314,044
2.6	0.55%	32,311
3.7	0.21%	12,522
3.3	0.12%	6,914
3.2	0.01%	754
3.8	0.00%	45
None	0.00%	39
Total		5,897,336

Source: pypinfo --start-date 2018-04-01 --end-date 2018-04-30 --percent --markdown urllib3 pyversion

[hugovk]

I see Requests 2.19 (2018-06-12) has now officially dropped support for Python 2.6 (psf/requests#4672) (and will remove the 2.6 code in Requests 3.0).

Any thoughts on dropping 2.6 in urllib3, before the full Python 2 removal?

---

Here's urllib3's July numbers:

python_version	percent	download_count
2.7	61.47%	6,259,611
3.6	19.08%	1,943,424
3.5	14.95%	1,522,056
3.4	2.65%	270,356
3.7	1.57%	160,020
2.6	0.23%	23,054
3.3	0.04%	3,798
3.8	0.00%	398
3.2	0.00%	393
None	0.00%	41
Total		10,183,151

[pquentin]

Yes, the maintainers would be happy to remove Python 2.6 now that pip dropped support. Is this something you'd like to work on?

[hugovk]

Yes, I can make a PR for it.

[sethmlarson]

I'm +1 to dropping 2.6 support. Our last release included features specifically for pip 9.x with the intent of dropping 2.6 after pip had done so.

[hugovk]

Please see PR #1429.

[sethmlarson]

Python 2.6 support has been dropped for the next release via #1429. Thanks @hugovk! Now to wait for another few years to drop Python 2.7.

[hugovk]

The general plan in 2016 was, hopefully sometime before 2020, for urllib3 v2.0 to drop Python 2 (along with a big backwards-compat breaking set of changes), and provide extended maintenance support for 1.x as long as pip needs it.

Is this still the plan?

Nine months out from Python 2's EOL (2020-01-01), is now a good time to make firmer plans?

---

PS pip is now warning (pypa/pip#6148):

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

[theacodes]

I think the general approach has been to wait until pip doesn't need it anymore. There's an active discussion going on here: https://discuss.python.org/t/packaging-and-python-2/662/95

[and-semakin]

According to https://pypistats.org/packages/urllib3 Python 2 still has 50%+ of downloads.

[sethmlarson]

@and-semakin There are no plans to drop Python 2 currently.

[hugovk]

I think the general approach has been to wait until pip doesn't need it anymore. There's an active discussion going on here: https://discuss.python.org/t/packaging-and-python-2/662/95

pip 20.3 (October 2020) will be the last version to support Python 2, and pip 21.0 (Jan 2021) will be the first version to drop support for Python 2.

pypa/pip#6148 (comment)

[pquentin]

I discussed with Seth, the current plan is:

When we're less than 10% and can drop 3.5 I'd be comfortable with a v2 that's 3.6+
1.x will receive bug fixes and security releases

[sethmlarson]

Looking at PyPI downloads for the most recent version 1.25.9 we're still a ways off from the "trigger point" with Python 2.7 at ~40%:

2	3
560656 38.3%	902301 61.7%

2.7	3.5	3.6	3.7	3.8	3.9
560656 38.3%	169307 11.6%	273659 18.7%	318376 21.8%	140698 9.6%	261 0.0%

cc @nateprewitt

[pradyunsg]

@sethmlarson I'd strongly recommend looking at downloads of the latest version (maybe ones released in the past month) instead of all versions. IMO it doesn't make sense to include folks who aren't upgrading to newer versions, when considering what to support in next versions.

For pip, the skew was 35-45% (all) vs 18%-20% (latest), and I wouldn't be surprised if it is similar for urllib3.

[sethmlarson]

Those numbers are from the most recent release (1.25.9). Totally agree that it doesn't make sense to count users who don't upgrade.

[pradyunsg]

Oh, I did a whooopsie and did not read carefully enough. Sorry for the noise! :)

I really hope a whole bunch of those Py2 PyPI downloads are from someone's cluster that will switch over to Py3 sometime soon.

[hugovk]

What tool/query did you both use to get numbers for only the latest version?

[sethmlarson]

For others here's the query I'm using for PyPI's dataset on June 10th (Wednesday):

SELECT
  SUBSTR(details.implementation.version, 1, 3) AS ver,
  COUNT(*) as dwn
FROM `the-psf.pypi.downloads20200610`
WHERE
  # Latest urllib3 version
  file.project = 'urllib3' AND
  file.version = '1.25.9' AND 
  # Filter out PyPy versions
  SUBSTR(details.implementation.version, 1, 1) <= '3' AND (
  # Filter out Python 2.6 or earlier
    SUBSTR(details.implementation.version, 1, 1) != '2' OR
    SUBSTR(details.implementation.version, 1, 3) = '2.7'
  ) AND (
  # Filter out Python 3.4 or earlier
    SUBSTR(details.implementation.version, 1, 1) != '3' OR
    SUBSTR(details.implementation.version, 3, 3) >= '5'
  )
GROUP BY ver
ORDER BY dwn DESC

This query results in the following:

Row	ver	dwn
1	2.7	824891
2	3.7	679098
3	3.6	618666
4	3.8	388414
5	3.5	283456
6	3.9	517

Running with only the major version:

Row	ver	dwn
1	3	1970151
2	2	824891

So the calculated percentage of Python 2.x is ~29.5%