How to process DNS lookup or client connection errors? #5179

Rozendantz · 2020-10-30T10:03:07Z

Rozendantz
Oct 30, 2020

🐞 Describe the bug

Connection Errors with known subdomains:

socket.gaierror: [Errno -2] Name or service not known
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host xxx.example.com:443 ssl:default [Name or service not known]

💡 To Reproduce

Scan a domain, any domain, using sublist3r. Pick a subdomain from the output of sublis3r and use with the following example code:

import aiohttp
import asyncio

async def main():

    async with aiohttp.ClientSession() as session:
        async with session.get('http://<domain from sublist3r>') as response:

            print("Status:", response.status)
            print("Content-type:", response.headers['content-type'])

            html = await response.text()
            print("Body:", html[:15], "...")

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

💡 Expected behavior

I expect to get a 404 status code

📋 Logs/tracebacks

future: <Task finished name='Task-8' coro=<get() done, defined at subforce.py:229> exception=ClientConnectorError(ConnectionKey(host='airviewstatus.resmed.com', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None), gaierror(-2, 'Name or service not known')) created at subforce.py:264>
source_traceback: Object created at (most recent call last):
  File "subforce.py", line 284, in <module>
    domains_loop.run_until_complete(iterate_domains())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1851, in _run_once
    handle._run()
  File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "subforce.py", line 264, in iterate_domains
    ret = domains_loop.create_task(get(domains_list))
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1001, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1001, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1001, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  [Previous line repeated 1 more time]
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 854, in _resolve_host
    await event.wait()
  File "/usr/lib/python3.8/site-packages/aiohttp/locks.py", line 38, in wait
    raise self._exc
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 854, in _resolve_host
    await event.wait()
  File "/usr/lib/python3.8/site-packages/aiohttp/locks.py", line 38, in wait
    raise self._exc
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 854, in _resolve_host
    await event.wait()
  File "/usr/lib/python3.8/site-packages/aiohttp/locks.py", line 38, in wait
    raise self._exc
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 867, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/usr/lib/python3.8/site-packages/aiohttp/resolver.py", line 31, in resolve
    infos = await self._loop.getaddrinfo(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 808, in _getaddrinfo_debug
    addrinfo = socket.getaddrinfo(host, port, family, type, proto, flags)
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "subforce.py", line 239, in get
    results = await asyncio.gather(*tasks) # gather task responses
  File "subforce.py", line 225, in fetch
    async with session.get(url) as response:
  File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 1124, in __aenter__
    self._resp = await self._coro
  File "/usr/lib/python3.8/site-packages/aiohttp/client.py", line 527, in _request
    conn = await self._connector.connect(
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 537, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 894, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/lib/python3.8/site-packages/aiohttp/connector.py", line 1013, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host airviewstatus.resmed.com:443 ssl:default [Name or service not known]
exception calling callback for <Future at 0x7f2fc8585940 state=finished returned list>

📋 Your version of the Python

$ python --version
Python 3.8.6

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
...Name: aiohttp
Version: 3.7.2
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /usr/lib/python3.8/site-packages
Requires: multidict, attrs, async-timeout, yarl, typing-extensions, chardet
Required-by:

$ python -m pip show multidict
Name: multidict
Version: 5.0.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/lib/python3.8/site-packages
Requires:
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.6.2
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /usr/lib/python3.8/site-packages
Requires: multidict, idna
Required-by: aiohttp

Linux
no proxy
client

Answered by webknjaz

Oct 30, 2020

Just use try/except to process aiohttp.ClientConnectionError per https://docs.aiohttp.org/en/stable/client_reference.html#connection-errors and that's it.

View full answer

webknjaz · 2020-10-30T11:10:29Z

webknjaz
Oct 30, 2020
Maintainer

socket.gaierror: [Errno -2] Name or service not known

That is a DNS lookup error, not 404.

0 replies

webknjaz · 2020-10-30T11:12:37Z

webknjaz
Oct 30, 2020
Maintainer

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host airviewstatus.resmed.com:443 ssl:default [Name or service not known]

There's nothing else to expect in such cases. If you cannot find a server, there's no way that server could send you 404.

0 replies

webknjaz · 2020-10-30T11:15:40Z

webknjaz
Oct 30, 2020
Maintainer

Just use try/except to process aiohttp.ClientConnectionError per https://docs.aiohttp.org/en/stable/client_reference.html#connection-errors and that's it.

5 replies

Rozendantz Oct 30, 2020
Author

Ah ok thanks for the pointer, I'll use a try/except.

Just by the by I've rewritten my code to use plain requests module with ThreadPoolExecutor and it's not giving me any errors on these domains. It fetches as expected, so they definitely exist and are responding.

Rozendantz Oct 30, 2020
Author

Nope that doesn't work either. Could you provide an example of how to use the exceptions?

Here is my code:

async def fetch(url, session):
    try:
        async with session.get(url) as response:
            resp = await response.read()
            return resp
    except:
        print("Connection Error")

async def get(domains_list):
    """Launch requests for all web pages."""
    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in domains_list:
            FQDM = "https://{domain}?".format(domain=url)
            print(FQDM)
            task = asyncio.ensure_future(fetch(FQDM, session))
            tasks.append(task) # create list of tasks
        results = await asyncio.gather(*tasks) # gather task responses

webknjaz Oct 30, 2020
Maintainer

What do you see exactly?

Rozendantz Oct 30, 2020
Author

With the above code I get the following output:

sys:1: RuntimeWarning: coroutine 'fetch' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Task was destroyed but it is pending!
task: <Task pending name='Task-24' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-25' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-26' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-27' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-28' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-29' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-30' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-31' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-32' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-33' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-34' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-35' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-36' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-37' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-38' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-39' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-40' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-41' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-42' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-43' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-44' coro=<fetch() running at subforce.py:228> cb=[gather.<locals>._done_callback() at /usr/lib/python3.8/asyncio/tasks.py:766]>
Task was destroyed but it is pending!
task: <Task pending name='Task-8' coro=<get() running at subforce.py:247> wait_for=<_GatheringFuture pending cb=[<TaskWakeupMethWrapper object at 0x7f7d78a8f8e0>()]>>
Unclosed client session

webknjaz Oct 30, 2020
Maintainer

That has nothing to do with aiohttp: you're using a low-level ensure_future API and it's your responsibility to await all the tasks you've created before the IO loop is shut down.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to process DNS lookup or client connection errors? #5179

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

How to process DNS lookup or client connection errors? #5179

Rozendantz Oct 30, 2020

Replies: 3 comments · 5 replies

webknjaz Oct 30, 2020 Maintainer

webknjaz Oct 30, 2020 Maintainer

webknjaz Oct 30, 2020 Maintainer

Rozendantz Oct 30, 2020 Author

Rozendantz Oct 30, 2020 Author

webknjaz Oct 30, 2020 Maintainer

Rozendantz Oct 30, 2020 Author

webknjaz Oct 30, 2020 Maintainer

Rozendantz
Oct 30, 2020

Replies: 3 comments 5 replies

webknjaz
Oct 30, 2020
Maintainer

webknjaz
Oct 30, 2020
Maintainer

webknjaz
Oct 30, 2020
Maintainer

Rozendantz Oct 30, 2020
Author

Rozendantz Oct 30, 2020
Author

webknjaz Oct 30, 2020
Maintainer

Rozendantz Oct 30, 2020
Author

webknjaz Oct 30, 2020
Maintainer