Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The aiohttp change with storage/dav/http #916

Closed
dilyanpalauzov opened this issue Jul 19, 2021 · 8 comments
Closed

The aiohttp change with storage/dav/http #916

dilyanpalauzov opened this issue Jul 19, 2021 · 8 comments
Labels
blocked cat: dependencies type: bug type: 3rd party

Comments

@dilyanpalauzov
Copy link
Contributor

With python 3.8.8+ and
aiohttp 3.7.4.post0
aiostream 0.4.3

I install vdirsyncer, version 0.18.1.dev44+g98fcd12. With this config file:

[general]
status_path = "status"

[pair google_download]
# From Google to the local file system
a = "google_remote"
b = "google_local"
collections = null
conflict_resolution = "a wins"

[storage google_remote]
type = "http"
url = "https://calendar.google.com/calendar/ical/aegee.eu_v5mn651imeqpvs87v4ln7hr1f4%40group.calendar.google.com/public/basic.ics"

[storage google_local]
type = "filesystem"
path = "google_local"
fileext = ".ics"

I call:

vdirsyncer -c config discover
google_remote:
google_local:
warning: Directory google_local does not exist.
warning: No collection null found for storage google_local.
Should vdirsyncer attempt to create it? [y/N]: y
Saved for google_download: collections = null


vdirsyncer -c config metasync
Metasyncing google_download

vdirsyncer -c config -vdebug sync
Syncing google_download
debug: ====================
debug: GET https://calendar.google.com/calendar/ical/aegee.eu_v5mn651imeqpvs87v4ln7hr1f4%40group.calendar.google.com/public/basic.ics
debug: {'User-Agent': 'vdirsyncer/0.18.1.dev44+g98fcd12'}
debug: None
debug: Sending request...
error: Unknown error occurred for google_download: tuple index out of range
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev44+g98fcd12-py3.8.egg/vdirsyncer/cli/tasks.py", line 72, in sync_collection
debug:     await sync.sync(
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev44+g98fcd12-py3.8.egg/vdirsyncer/sync/__init__.py", line 144, in sync
debug:     a_nonempty = await a_info.prepare_new_status()
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev44+g98fcd12-py3.8.egg/vdirsyncer/sync/__init__.py", line 48, in prepare_new_status
debug:     async for href, etag in self.storage.list():
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev44+g98fcd12-py3.8.egg/vdirsyncer/storage/http.py", line 68, in list
debug:     r = await request(
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev44+g98fcd12-py3.8.egg/vdirsyncer/http.py", line 149, in request
debug:     r = await r
debug:   File "/usr/local/lib/python3.8/site-packages/aiohttp-3.7.4.post0-py3.8-linux-x86_64.egg/aiohttp/client.py", line 520, in _request
debug:     conn = await self._connector.connect(
debug:   File "/usr/local/lib/python3.8/site-packages/aiohttp-3.7.4.post0-py3.8-linux-x86_64.egg/aiohttp/connector.py", line 535, in connect
debug:     proto = await self._create_connection(req, traces, timeout)
debug:   File "/usr/local/lib/python3.8/site-packages/aiohttp-3.7.4.post0-py3.8-linux-x86_64.egg/aiohttp/connector.py", line 892, in _create_connection
debug:     _, proto = await self._create_direct_connection(req, traces, timeout)
debug:   File "/usr/local/lib/python3.8/site-packages/aiohttp-3.7.4.post0-py3.8-linux-x86_64.egg/aiohttp/connector.py", line 999, in _create_direct_connection
debug:     hosts = await asyncio.shield(host_resolved)
debug:   File "/usr/local/lib/python3.8/site-packages/aiohttp-3.7.4.post0-py3.8-linux-x86_64.egg/aiohttp/connector.py", line 865, in _resolve_host
debug:     addrs = await self._resolver.resolve(host, port, family=self._family)
debug:   File "/usr/local/lib/python3.8/site-packages/aiohttp-3.7.4.post0-py3.8-linux-x86_64.egg/aiohttp/resolver.py", line 41, in resolve
debug:     if family == socket.AF_INET6 and address[3]:  # type: ignore

However, if I try the same on a different computer with python 3.9.6, again with

aiohttp                       3.7.4.post0
aiostream                     0.4.3

There is no such error, with the same vdirsyncer code (0.18.1.dev44+g98fcd12).

My understanding is, that vdirsyncer is supposed to work under python 3.8.
@dilyanpalauzov
Copy link
Contributor Author

if family == socket.AF_INET6 and address[3]:

The system has no IPv6 support configured, but the kernel does know what IPv6 is.

@WhyNotHugo
Copy link
Member

Should definitely work on Python 3.8. What distribution are you using?

@dilyanpalauzov
Copy link
Contributor Author

I use Linux from Scratch. With this change to aiohttp in the 3.8 branch

diff --git a/aiohttp/resolver.py b/aiohttp/resolver.py
index 2161c8fa..d20bf818 100644
--- a/aiohttp/resolver.py
+++ b/aiohttp/resolver.py
@@ -38,6 +38,7 @@ class ThreadedResolver(AbstractResolver):
 
         hosts = []
         for family, _, proto, _, address in infos:
+            print(family, address)
             if family == socket.AF_INET6 and address[3]:  # type: ignore[misc]
                 # This is essential for link-local IPv6 addresses.
                 # LL IPv6 is a VERY rare case. Strictly speaking, we should use

vdirsyncer prints:

$ vdirsyncer -vdebug -c config sync google_download
Syncing google_download
debug: ====================
debug: GET https://calendar.google.com/calendar/ical/aegee.eu_v5mn651imeqpvs87v4ln7hr1f4%40group.calendar.google.com/public/basic.ics
debug: {'User-Agent': 'vdirsyncer/0.18.1.dev49+gfcc6130'}
debug: None
debug: Sending request...
AddressFamily.AF_INET ('142.250.185.238', 443)
AddressFamily.AF_INET6 (10, b'\x01\xbb\x00\x00\x00\x00*\x00\x14P@\x01\x08\x13')
error: Unknown error occurred for google_download: tuple index out of range
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev49+gfcc6130-py3.8.egg/vdirsyncer/cli/tasks.py", line 72, in sync_collection
debug:     await sync.sync(
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev49+gfcc6130-py3.8.egg/vdirsyncer/sync/__init__.py", line 144, in sync
debug:     a_nonempty = await a_info.prepare_new_status()
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev49+gfcc6130-py3.8.egg/vdirsyncer/sync/__init__.py", line 48, in prepare_new_status
debug:     async for href, etag in self.storage.list():
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev49+gfcc6130-py3.8.egg/vdirsyncer/storage/http.py", line 68, in list
debug:     r = await request(
debug:   File "/usr/local/lib/python3.8/site-packages/vdirsyncer-0.18.1.dev49+gfcc6130-py3.8.egg/vdirsyncer/http.py", line 145, in request
debug:     r = await r
debug:   File "/git/aiohttp/aiohttp/client.py", line 518, in _request
debug:     conn = await self._connector.connect(
debug:   File "/git/aiohttp/aiohttp/connector.py", line 543, in connect
debug:     proto = await self._create_connection(req, traces, timeout)
debug:   File "/git/aiohttp/aiohttp/connector.py", line 906, in _create_connection
debug:     _, proto = await self._create_direct_connection(req, traces, timeout)
debug:   File "/git/aiohttp/aiohttp/connector.py", line 1013, in _create_direct_connection
debug:     hosts = await asyncio.shield(host_resolved)
debug:   File "/git/aiohttp/aiohttp/connector.py", line 879, in _resolve_host
debug:     addrs = await self._resolver.resolve(host, port, family=self._family)
debug:   File "/git/aiohttp/aiohttp/resolver.py", line 42, in resolve
debug:     if family == socket.AF_INET6 and address[3]:  # type: ignore[misc]

address[3] does not exist.

@dilyanpalauzov
Copy link
Contributor Author

I fixed aiohttp with

diff --git a/aiohttp/resolver.py b/aiohttp/resolver.py
index 2161c8fa..4e57b319 100644
--- a/aiohttp/resolver.py
+++ b/aiohttp/resolver.py
@@ -38,16 +38,7 @@ class ThreadedResolver(AbstractResolver):
 
         hosts = []
         for family, _, proto, _, address in infos:
-            if family == socket.AF_INET6 and address[3]:  # type: ignore[misc]
-                # This is essential for link-local IPv6 addresses.
-                # LL IPv6 is a VERY rare case. Strictly speaking, we should use
-                # getnameinfo() unconditionally, but performance makes sense.
-                host, _port = socket.getnameinfo(
-                    address, socket.NI_NUMERICHOST | socket.NI_NUMERICSERV
-                )
-                port = int(_port)
-            else:
-                host, port = address[:2]
+            host, port = address[:2]
             hosts.append(
                 {
                     "hostname": hostname,

@dilyanpalauzov dilyanpalauzov mentioned this issue Jul 19, 2021
Closed
1 task
@dilyanpalauzov
Copy link
Contributor Author

See also aio-libs/aiohttp#5901.

@WhyNotHugo
Copy link
Member

Looks like a bug in aiohttp, so not much I can do here.

When a release with a fix comes out, I'll pin that as a minimal version.

@WhyNotHugo
Copy link
Member

Current upstream: aio-libs/aiohttp#5156

@dilyanpalauzov
Copy link
Contributor Author

aiohttp 3.8.0 remedies the problem.

WhyNotHugo pushed a commit that referenced this issue Nov 2, 2021
Require aiohttp >= 3.8.0
961203e 
See #916 (comment)
See aio-libs/aiohttp#5156
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocked cat: dependencies type: bug type: 3rd party
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@WhyNotHugo @dilyanpalauzov