Connection Reset by peer error using aiohttp and asyncio

[parth995]

Describe the bug

I am getting aiohttp.client_exceptions.ClientConnectorError upon using aiohttp and asyncio together. I am gathering all tasks using one session by aiohttp and getting responses.

To Reproduce

import asyncio
import aiohttp
import requests
from aiohttp import ClientSession

def get_tasks(session):
      tasks = []
      for item in ips:
      tasks.append(session.get(base_url, ssl=False))
      return tasks

async def push_ips():
      async with aiohttp.ClientSession() as session:
            tasks = get_tasks(session)
            responses = await asyncio.gather(*tasks)
            for response in responses:
                     print(response)

asyncio.run(push_ips())

Expected behavior

Tasks executed in queue with status code 200.

Logs/tracebacks

Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/aiohttp/connector.py”, line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File “/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py”, line 1081, in create_connection
transport, protocol = await self._create_connection_transport(
File “/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py”, line 1111, in _create_connection_transport
await waiter
File “/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/selector_events.py”, line 856, in _read_ready__data_received
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 54] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/async_test2.py”, line 59, in
asyncio.run(push_ips())
File “/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py”, line 44, in run
return loop.run_until_complete(main)
File “/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py”, line 642, in run_until_complete
return future.result()
File “/async_test2.py”, line 54, in push_ips
responses = await asyncio.gather(*tasks)
File “/usr/local/lib/python3.9/site-packages/aiohttp/client.py”, line 1104, in throw
self._coro.throw(arg)
File “/usr/local/lib/python3.9/site-packages/aiohttp/client.py”, line 520, in _request
conn = await self._connector.connect(
File “/usr/local/lib/python3.9/site-packages/aiohttp/connector.py”, line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File “/usr/local/lib/python3.9/site-packages/aiohttp/connector.py”, line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File “/usr/local/lib/python3.9/site-packages/aiohttp/connector.py”, line 1051, in _create_direct_connection
raise last_exc
File “/usr/local/lib/python3.9/site-packages/aiohttp/connector.py”, line 1020, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File “/usr/local/lib/python3.9/site-packages/aiohttp/connector.py”, line 975, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host xxxxxxxxxxxxxx ssl:False [Connection reset by peer]

Python Version

$ python --version
3.9.6

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.4.post0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /usr/local/lib/python3.9/site-packages
Requires: async-timeout, multidict, attrs, typing-extensions, chardet, yarl
Required-by:

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 5.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/local/lib/python3.9/site-packages
Requires: 
Required-by: yarl, aiohttp

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/local/lib/python3.9/site-packages
Requires: multidict, idna
Required-by: aiohttp

OS

macOS - Darwin Kernel Version 19.6.0

Related component

Client

Additional context

No response

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[webknjaz]

[Errno 54] Connection reset by peer

This means that the server drops connections on the TCP level. Maybe because you're spamming them. Verify that with the server side. There's nothing we can do.

You could wrap tasks with try/except catching ClientConnectorError in order to ignore or retry after some backoff but that's it.

[parth995]

Thanks mate for prompt response. Let me check further on this

[bero-dev]

[Errno 54] Connection reset by peer

This means that the server drops connections on the TCP level. Maybe because you're spamming them. Verify that with the server side. There's nothing we can do.

You could wrap tasks with try/except catching ClientConnectorError in order to ignore or retry after some backoff but that's it.

Is there a specific cooldown for this? I'm having this issue for the first time now. It's a loop which does a specific task every 5 minutes.
Versions are similar to the one created this thread.

[a2liro]

I have same problem... any solution?

[michaelurban]

I'm getting a combination of aiohttp.client_exceptions.ServerDisconnectedError and aiohttp.client_exceptions.ClientOSError.

I control both the client and server side.

Underlying connections are routinely closed for a variety of reasons. Shouldn't the connection pool handle server disconnects and transparently provide the client with a usable connection?

Is this an effective workaround or is there a possibility of request duplication:

max_retries = 5
attempt = 0

while True:
    try:
        # `self._http_client_session` is initialized with the process.
        async with self._http_client_session.request(...) as upstream_response:
            ...    
            return
    except (
        aiohttp.ClientOSError,
        aiohttp.ServerDisconnectedError,
    ):
        if attempt < max_retries:
            attempt += 1
        else:
            raise

[zengzzzzz]

got it， thanks bro。