responses 0.21.0 broke mypy check

[ondratu]

Describe the bug

We use response in our tests, and from responses 0.21.0 we've got this error:

$mypy tests/test_download.py 
venv/lib/python3.10/site-packages/mypy/typeshed/stdlib/unittest/mock.pyi: error: Source file found twice under different module names: "unittest.mock" and "mypy.typeshed.stdlib.unittest.mock"
Found 1 error in 1 file (errors prevented further checking)

Returning mypy version did not resolve problem, but returning responses to 0.20.0 does.

I try to find problem in your changes, and it could be probably this line in init.py:

from mypy.typeshed.stdlib.unittest.mock import _patcher as _mock_patcher

Additional context

No response

Version of responses

0.21.0

Steps to Reproduce

# content of crash.py

import responses
import unittest.mock

and than run

mypy crash.py

Expected Result

Success: no issues found in 1 source file

Actual Result

venv/lib/python3.10/site-packages/mypy/typeshed/stdlib/unittest/mock.pyi: error: Source file found twice under different module names: "unittest.mock" and "mypy.typeshed.stdlib.unittest.mock"
Found 1 error in 1 file (errors prevented further checking)

[markstory]

I'm not able to reproduce the problem you're having

ᐉ pyenv install 3.10.1
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.1.tar.xz...
-> https://www.python.org/ftp/python/3.10.1/Python-3.10.1.tar.xz
Installing Python-3.10.1...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.1 to /Users/markstory/.pyenv/versions/3.10.1

ᐉ pyenv local 3.10.1
ᐉ pip install responses
Collecting responses
  Downloading responses-0.21.0-py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 4.0 MB/s
Collecting requests<3.0,>=2.0
  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
     |████████████████████████████████| 63 kB 8.5 MB/s
Collecting urllib3>=1.25.10
  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 16.2 MB/s
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2022.5.18.1-py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 30.1 MB/s
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, responses
Successfully installed certifi-2022.5.18.1 charset-normalizer-2.0.12 idna-3.3 requests-2.27.1 responses-0.21.0 urllib3-1.26.9
WARNING: You are using pip version 21.2.4; however, version 22.1.1 is available.
You should consider upgrading via the '/Users/markstory/.pyenv/versions/3.10.1/bin/python3.10 -m pip install --upgrade pip' command.
ᐉ pip install mypy
Collecting mypy
  Downloading mypy-0.960-cp310-cp310-macosx_11_0_arm64.whl (10.2 MB)
     |████████████████████████████████| 10.2 MB 5.1 MB/s
Collecting typing-extensions>=3.10
  Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Collecting tomli>=1.1.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting mypy-extensions>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Installing collected packages: typing-extensions, tomli, mypy-extensions, mypy
Successfully installed mypy-0.960 mypy-extensions-0.4.3 tomli-2.0.1 typing-extensions-4.2.0
WARNING: You are using pip version 21.2.4; however, version 22.1.1 is available.
You should consider upgrading via the '/Users/markstory/.pyenv/versions/3.10.1/bin/python3.10 -m pip install --upgrade pip' command.
ᐉ cp ../responses/crash.py .
ᐉ mypy crash.py
Success: no issues found in 1 source file
ᐉ cat crash.py
# content of crash.py

import responses
import unittest.mock

[ondratu]

You are right, i try to run in different place and i got your behavior. So i found that i have setup.cfg in same place with next lines

[mypy]
namespace_packages = true

which is similar to

mypy --namespace-packages crash.py

And that you will get same problem:

venv/lib/python3.10/site-packages/mypy/typeshed/stdlib/unittest/mock.pyi: error: Source file found twice under different module names: "unittest.mock" and "mypy.typeshed.stdlib.unittest.mock"
Found 1 error in 1 file (errors prevented further checking)

[markstory]

We're using the typeshed package to access that type because the unitest.mock module doesn't expose an importable type for the return of mock.patch().

[ondratu]

I did not try it, resp. when i try to call mypy responses, there are too many errors... but... what about use unittest.mock._patch instead of mypy.typeshed.stdlib.unittest.mock._patcher ?

And if I remember right, there is not need to import type, when you using type name..

[beliaev-maksim]

@ondratu
I do not think python 3.7 has stub file for mock._patch

I did not check if 3.10 has

side question, do you really create namespace package?
because to my knowledge this one tries to resolve ambiguous paths and this might result this issue

[ondratu]

Namespace packages are perfect for ... packages in namespace, for example when you have core / library / application and many plugins. There are two choices:

• application_plugin (in /application_plugin)
• application.plugin (in /application/plugin/)

When you have many plugins... namespace are great, and that is why they are supported ;)

[beliaev-maksim]

yes, I understand the purpose but also have seen that sometimes people misuse them :)

anyway, I need to check the difference in unittest stubs in 3.7 and 3.10

[beliaev-maksim]

I think it comes from mypy. Looks like namespace feature is incomplete.
reading the thread and pinning mypy==0.770 solves this issue, but then it causes other issues since we were developing for mypy 0.900+

There are no default stubs for unittest.mock, so, we have to use external stubs.

@markstory
I do not think we can propose a solution here

[sodul]

I can confirm that I reproduced the same issue. We are using python 3.10.5, mypy 0.961 and upgrading responses from 0.20.0 to 0.21.0 got mypy confused.

[beliaev-maksim]

@sodul
Do you also use namespace packages?

[sodul]

@beliaev-maksim yes our mypy.ini does contain namespace_packages=True.

[beliaev-maksim]

@ondratu @sodul
can you check if https://github.com/beliaev-maksim/responses/tree/mbeliaev/mypy fixes your issue?

[sodul]

pip uninstall responses && pip install 'https://github.com/beliaev-maksim/responses/archive/mbeliaev/mypy.zip#egg=responses'

seems to fix the problem.

I rolled back to responses==0.21.0 and was able to reproduce the issue: pip uninstall responses && pip install responses==0.21.0.

Note that an explicit uninstall helps with the internal version tracking of pip.

[ancalita]

@markstory Experiencing the same issue when trying to upgrade responses in a Rasa private repo - could you please let us know about timeline for a new release (micro or minor) now that the PR with the fix got merged?

[fredrikaverpil]

It seems this issue shouldn't be closed, or should we track this elsewhere?

[beliaev-maksim]

@fredrikaverpil
Can you reproduce it on the latest master?

[fredrikaverpil]

@beliaev-maksim

$ mypy --version
mypy 0.971 (compiled: yes)

With responses = {git = "https://github.com/getsentry/responses.git", rev = "master"} in pyproject.toml:

$ poetry update responses 
Updating dependencies
Resolving dependencies... (17.8s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating responses (0.20.0 -> 0.21.0 1db612d)

$ mypy
Success: no issues found in 232 source files

Looks like it's fixed in master! 🎉

[kawa-marcin]

Thank you so much for your work on that and fix 💌 Sorry for a dummy question here but I couldn't find this information in the docs (about how often the new version of the package is released) ... Is there a plan to release this fix as the new version of requests, and if so, what's the release schedule so the proper package version can be pinned instead of pinning the master branch in the requirements.

[beliaev-maksim]

Please track #576

[marklitle]

I just fixed this issue
For my case:
Some python file imports a same python file,
some use relative path while some use absolute path
This cause mypy confused when it searches the hash table
Check the imports and you don't have to delete init.py