Release 0.790 planning

[JukkaL]

I plan to release mypy 0.790 around Aug 25 and cut the release branch this Thu (Aug 13).

Please add comments to this issue if you have any PR you'd like to be included. Ideally those PRs should be landed before the release branch is cut, to minimize cherry-picking.

(This will hopefully be the final major release with bundled monolithic typeshed. 0.800 should have third-party package stubs available from PyPI.)

[JukkaL]

I'm encountering what looks like a hard crash (no traceback) on Linux with Python 3.6 when running the latest master against Dropbox internal code.

It started happening after 88eb84e was merged. Looking at that PR, I don't see anything particularly suspicious, but it's doing some pointer casting which might plausibly be a problem. If I can't fix the crash soon, I might cut the release branch from an earlier commit or wait a bit longer until I cut the release branch.

[JukkaL]

I narrowed down the segfault to mypyc/mypyc#756. It's a release-blocker, unless we cut the release branch from an earlier commit and do some cherry-picks.

[msullivan]

Is the branch cut happening today? I've been seeing infinite loops with --follow-imports=normal on https://github.com/edgedb/ and will try to investigate and get a PR up today.

[TH3CHARLie]

Is the branch cut happening today? I've been seeing infinite loops with --follow-imports=normal on https://github.com/edgedb/ and will try to investigate and get a PR up today.

I think it will happen today since the fix for mypyc/mypyc#756 would be then included and no cherry-picking is needed.

[msullivan]

I'd like #9302 to make it into the release

[hauntsaninja]

I'm late to the party, but it'd be nice if this typeshed sync #9317 containing python/typeshed#4447 made it in

[TH3CHARLie]

I'm late to the party, but it'd be nice if this typeshed sync #9317 containing python/typeshed#4447 made it in

according to the branches page, the release 0.790 branch was already cut last Friday.

[JukkaL]

The release will be delayed by at least a few days, since I didn't manage to update Dropbox internal mypy version to the release branch last week, and this week I'm on vacation. I want at least 3 days of internal testing at Dropbox before a release.

@hauntsaninja I'll need to make some other typeshed fixes as well before the release, so there should be no problems including your fix as well.

[msullivan]

Any updates on release schedule?

[gvanrossum]

I imagine the internal testing might have been prolonged due to the flurry of typeshed changes due to some folks at Microsoft submitting a truckload of new packages and tightening many existing packages.

[JukkaL]

Yeah, the release is delayed because of the various typeshed changes that either require fixes in typeshed or fixes to our internal codebases. Also some of them exposed mypy bugs that we may want to fix before the release.

I already used all the time I had reserved for preparing for the release, so I'll have to scramble a few hours here and there to finish the release, unfortunately. (But I want to get it out ASAP.)

I'll post an update once we've started testing the release branch internally.

[ethanhs]

Since Python 3.5 is going to be sunsetting in 11 days, may I recommend we make this the last release to support running mypy with Python 3.5 as well?

[JukkaL]

Update: I have a bit too much stuff going on right now to make progress with the release. I'll get back to this as soon as things calm down a bit.

[JukkaL]

@ethanhs I think that we can drop support for running with Python 3.5 soon. Pip will drop support in January 2021 (pypa/pip#8778), and that might be a reasonable timeframe for us as well to give users some time to upgrade.

[srittau]

According to https://www.python.org/downloads/release/python-3510/ Python 3.5 reaches end of life today. I will suggest to drop it from typeshed after the next mypy release.

[JelleZijlstra]

@JukkaL let me know if there's anything people outside Dropbox can do to help.

Are you still planning to use the current 0.790 release branch? Given the delay it might be better to cut the branch from master again, even if that requires a bit more testing.

[JukkaL]

There was some extra friction from broken wheel builds, but they seem to work again now. I'll try to update the internal mypy version used at Dropbox tomorrow. I'll post a new release date estimate once the update has been successful.

@JelleZijlstra Cutting from master may result in extra work, since there may be additional typeshed issues that need to be fixed. I'll try to move forward with the current 0.790 branch, but if I hit issues, I'll consider cutting the branch again.

[hauntsaninja]

Let us know how we can help!

For whatever it's worth, I ran mypy_primer on HEAD vs release-0.790. Results:

Collected using python -m primer --old release-0.790 -o concise

Fixed (presumably) false positives:

Due to improvements that have been made to the newly introduced stubs for markdown, we fix about 40 errors in a project that makes heavy use of markdown (not shown below). Other fixes:

- src/werkzeug/utils.py:646: error: Only @runtime_checkable protocols can be used with instance and class checks
+ src/werkzeug/security.py:165: error: unused 'type: ignore' comment
+ src/werkzeug/security.py:170: error: unused 'type: ignore' comment

- src/black/__init__.py:265:75: error: Returning Any from function declared to return "_SupportsLessThan"

- paasta_tools/mesos/master.py:150: error: Argument "delay" to "retry" has incompatible type "float"; expected "int"

+ sphinx/builders/html/__init__.py:94: error: unused 'type: ignore' comment
+ sphinx/builders/html/__init__.py:117: error: unused 'type: ignore' comment

+ homeassistant/core.py:163: error: unused 'type: ignore' comment
+ homeassistant/core.py:528: error: unused 'type: ignore' comment
+ homeassistant/helpers/location.py:24: error: unused 'type: ignore' comment

New errors:

+ dedupe/datamodel.py:12: error: Incompatible types in assignment (expression has type "bool", variable has type "Union[MetaPathFinder, PathEntryFinder]")

+ isort/output.py:561: error: "_LineWithComments" has no attribute "comments"

+ homeassistant/helpers/device_registry.py:568: error: Unsupported operand types for - ("Set[str]" and "Set[Optional[str]]")

+ tests/test_user_registry.py:93: error: Dict entry 0 has incompatible type "str": "User"; expected "str": "Sequence[Union[str, int]]"

+ zerver/management/commands/check_redis.py:52: error: Incompatible types in assignment (expression has type "Callable[[str, int], bool]", variable has type "Optional[Callable[[str, int], None]]")  [assignment]

+ zerver/lib/markdown/__init__.py:1787: error: Return type "Iterable[str]" of "run" incompatible with return type "List[str]" in supertype "Preprocessor"  [override]ned]
+ zerver/lib/markdown/__init__.py:1904: error: Module has no attribute "BlockParser"  [attr-defined]
+ zerver/lib/markdown/__init__.py:2020: error: "Registry" has no attribute "blockprocessors"  [attr-defined]

The first 2 are from typeshed changes (and are correct, although if mypy allowed redefining _ the first one would go away). The last 3 markdown errors replace ~40 old ones (once python/typeshed#4588 is merged). Haven't looked into the ones in between

We no longer crash on this project (and so it reports some "new" errors):

- ./suggestionbox/core.py:38: error: INTERNAL ERROR -- Please try using mypy master on Github:
- https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
- If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
- version: 0.790+dev.498769b5531630016341cb55ead6df3682240368
- ./suggestionbox/core.py:38: : note: please use --show-traceback to print a traceback when reporting a bug
- /private/tmp/mypy_primer/old_mypy/venv/lib/python3.8/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:116: note: "__init_subclass__" of "object" defined here
+ /private/tmp/mypy_primer/new_mypy/venv/lib/python3.8/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:117: note: "__init_subclass__" of "object" defined here
+ anticommandspam/core.py:130: error: unused 'type: ignore' comment
+ [..."new" errors...]

Other differences:

About 20 counts of an improved error message due to this change renaming _LT to _SupportsLessThanT: python/typeshed#4579

- src/aiortc/mediastreams.py:22: note:     def __init__(self, x: Union[str, bytes, SupportsInt, _SupportsIndex] = ...) -> int
+ src/aiortc/mediastreams.py:22: note:     def __new__(cls, x: Union[str, SupportsInt, _SupportsIndex] = ...) -> int

- dragonchain/lib/database/redis_utest.py:114:9: error: "Callable[[VarArg(Union[str, bytes])], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:114:9: error: "Callable[[VarArg(Union[str, bytes])], int]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:138:9: error: "Callable[[VarArg(Union[str, bytes])], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:138:9: error: "Callable[[VarArg(Union[str, bytes])], int]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Any, Any], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:158:9: error: "Callable[[Union[str, bytes], int], Optional[bytes]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:166:9: error: "Callable[[Any, Any, Any], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:166:9: error: "Callable[[Union[str, bytes], int, int], bool]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:174:9: error: "Callable[[Any, VarArg(Any)], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:174:9: error: "Callable[[Union[str, bytes], VarArg(Union[bytes, float, int, str])], int]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:178:9: error: "Callable[[Any, Any], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:178:9: error: "Callable[[Union[str, bytes], Union[bytes, float, int, str]], bool]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Any, Any, Any], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:190:9: error: "Callable[[Union[str, bytes], int, int], List[bytes]]" has no attribute "assert_called_once_with"
- dragonchain/lib/database/redis_utest.py:198:9: error: "Callable[[Any], Any]" has no attribute "assert_called_once_with"
+ dragonchain/lib/database/redis_utest.py:198:9: error: "Callable[[Union[str, bytes]], int]" has no attribute "assert_called_once_with"

I also wouldn't complain if you found the time to cherry pick #9479 and #9495. Both are mypy crashes that have been reported multiple times (the first one is surfaced by mypy_primer as well).

[JukkaL]

@hauntsaninja Thanks for the detailed analysis! I'll try running the latest master against our internal repos; if the number of new errors is small enough, I can cut the release branch again from master. (The current release branch generated about 2000 errors initially, I think, but hopefully things are less bad now.)

[JukkaL]

At least python/typeshed#4591 blocks the use of master for the release.

[JukkaL]

@hauntsaninja Cherry-picked #9479 and #9495.

[JukkaL]

The new ETA for the release is around Fri Oct 9, if no major new issues are found. We are now testing the release branch internally at Dropbox.

I'll write a postmortem with ideas about how to make the release process smoother after the release is out.

[JukkaL]

Cherry-picked #9538 and #9539, as these address issues with a feature introduced in this release.

[JukkaL]

Mypy 0.790 is finally out: https://mypy-lang.blogspot.com/2020/10/mypy-0790-released.html

[TH3CHARLie]

thanks for the hard work! @JukkaL

[hauntsaninja]

@JukkaL would it be possible to make a 0.791 release including #9587? (mypyc/mypy_mypyc-wheels#11 (comment) and #9584)

[JukkaL]

Sure, I can make the 0.791 release. I had a cold last week and am only now catching up with what's been happening.