Python 3.8 support

[thkukuk]

With python 3.8 there are some errors in the test suite:
FAILED tests/connection_test.py::test_connect_tcp_timeout[redis_v5.0.7] - Typ...
FAILED tests/connection_test.py::test_connect_unixsocket_timeout[redis_v5.0.7]
FAILED tests/pool_test.py::test_create_connection_timeout[redis_v5.0.7] - Typ...
FAILED tests/pool_test.py::test_pool_idle_close[redis_v5.0.7] - assert [('aio...

[honglei]

CentOS8.1/redis 5.0.3

[root@localhost aioredis-1.3.1]# /opt/python38/bin/pytest
=================================================================================================== test session starts ===================================================================================================
platform linux -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /root/python/aioredis-1.3.1, inifile: setup.cfg
plugins: hypothesis-5.6.0, cov-2.8.1, forked-1.1.3, xdist-1.31.0
collected 799 items

tests/coerced_keys_dict_test.py ...                                                                                                                                                                                 [  0%]
tests/connection_commands_test.py ..................                                                                                                                                                                [  2%]
tests/connection_test.py ...F..F..............................                                                                                                                                                      [  7%]
tests/generic_commands_test.py ................................................................                                                                                                                     [ 15%]
tests/geo_commands_test.py ......................                                                                                                                                                                   [ 18%]
tests/hash_commands_test.py ....................................                                                                                                                                                    [ 22%]
tests/hyperloglog_commands_test.py ..........                                                                                                                                                                       [ 23%]
tests/integration_test.py ...x....                                                                                                                                                                                  [ 24%]
tests/list_commands_test.py ........................................                                                                                                                                                [ 29%]
tests/pool_test.py ..........F..............X...........F...                                                                                                                                                        [ 34%]
tests/pubsub_commands_test.py ................................                                                                                                                                                      [ 38%]
tests/pubsub_receiver_test.py ..........                                                                                                                                                                            [ 40%]
tests/scripting_commands_test.py ............                                                                                                                                                                       [ 41%]
tests/sentinel_commands_test.py ....................                                                                                                                                                                [ 44%]
tests/sentinel_failover_test.py ..Xxx.                                                                                                                                                                              [ 44%]
tests/server_commands_test.py ..............................................                                                                                                                                        [ 50%]
tests/set_commands_test.py ..................................                                                                                                                                                       [ 54%]
tests/sorted_set_commands_test.py ..............................................................                                                                                                                    [ 62%]
tests/ssl_test.py ssss                                                                                                                                                                                              [ 63%]
tests/stream_commands_test.py ............................................................                                                                                                                          [ 70%]
tests/string_commands_test.py ..................................................................                                                                                                                    [ 78%]
tests/task_cancellation_test.py .                                                                                                                                                                                   [ 79%]
tests/transaction_commands_test.py ..............................                                                                                                                                                   [ 82%]
tests/encode_command_test.py .......                                                                                                                                                                                [ 83%]
tests/errors_test.py ...                                                                                                                                                                                            [ 84%]
tests/locks_test.py .                                                                                                                                                                                               [ 84%]
tests/multi_exec_test.py .                                                                                                                                                                                          [ 84%]
tests/parse_url_test.py ...............................................................                                                                                                                             [ 92%]
tests/pubsub_receiver_test.py .....s.                                                                                                                                                                               [ 93%]
tests/pyreader_test.py ...........................................x                                                                                                                                                 [ 98%]
tests/stream_commands_test.py ...                                                                                                                                                                                   [ 98%]
tests/stream_test.py ........                                                                                                                                                                                       [100%]

======================================================================================================== FAILURES =========================================================================================================
_________________________________________________________________________________________ test_connect_tcp_timeout[redis_v5.0.3] __________________________________________________________________________________________

request = <FixtureRequest for <Function test_connect_tcp_timeout[redis_v5.0.3]>>, create_connection = <function create_connection.<locals>.f at 0x7f7a08715af0>
server = RedisServer(name='A', tcp_address=TCPAddress(host='localhost', port=55631), unixsocket='/tmp/aioredis.55631.sock', version=(5, 0, 3), password=None)

    async def test_connect_tcp_timeout(request, create_connection, server):
        with patch('aioredis.connection.open_connection') as open_conn_mock:
            open_conn_mock.side_effect = lambda *a, **kw: asyncio.sleep(0.2)
            with pytest.raises(asyncio.TimeoutError):
>               await create_connection(server.tcp_address, timeout=0.1)

tests/connection_test.py:64:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/conftest.py:68: in f
    conn = await aioredis.create_connection(*args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = TCPAddress(host='localhost', port=55631)

    async def create_connection(address, *, db=None, password=None, ssl=None,
                                encoding=None, parser=None, loop=None,
                                timeout=None, connection_cls=None):
        """Creates redis connection.

        Opens connection to Redis server specified by address argument.
        Address argument can be one of the following:
        * A tuple representing (host, port) pair for TCP connections;
        * A string representing either Redis URI or unix domain socket path.

        SSL argument is passed through to asyncio.create_connection.
        By default SSL/TLS is not used.

        By default any timeout is applied at the connection stage, however
        you can set a limitted time used trying to open a connection via
        the `timeout` Kw.

        Encoding argument can be used to decode byte-replies to strings.
        By default no decoding is done.

        Parser parameter can be used to pass custom Redis protocol parser class.
        By default hiredis.Reader is used (unless it is missing or platform
        is not CPython).

        Return value is RedisConnection instance or a connection_cls if it is
        given.

        This function is a coroutine.
        """
        assert isinstance(address, (tuple, list, str)), "tuple or str expected"
        if isinstance(address, str):
            address, options = parse_url(address)
            logger.debug("Parsed Redis URI %r", address)
            db = options.setdefault('db', db)
            password = options.setdefault('password', password)
            encoding = options.setdefault('encoding', encoding)
            timeout = options.setdefault('timeout', timeout)
            if 'ssl' in options:
                assert options['ssl'] or (not options['ssl'] and not ssl), (
                    "Conflicting ssl options are set", options['ssl'], ssl)
                ssl = ssl or options['ssl']

        if timeout is not None and timeout <= 0:
            raise ValueError("Timeout has to be None or a number greater than 0")

        if connection_cls:
            assert issubclass(connection_cls, AbcConnection),\
                    "connection_class does not meet the AbcConnection contract"
            cls = connection_cls
        else:
            cls = RedisConnection

        if loop is not None and sys.version_info >= (3, 8, 0):
            warnings.warn("The loop argument is deprecated",
                          DeprecationWarning)

        if isinstance(address, (list, tuple)):
            host, port = address
            logger.debug("Creating tcp connection to %r", address)
>           reader, writer = await asyncio.wait_for(open_connection(
                host, port, limit=MAX_CHUNK_SIZE, ssl=ssl),
                timeout)
E           TypeError: cannot unpack non-iterable coroutine object

/opt/python38/lib/python3.8/site-packages/aioredis/connection.py:111: TypeError
______________________________________________________________________________________ test_connect_unixsocket_timeout[redis_v5.0.3] ______________________________________________________________________________________

create_connection = <function create_connection.<locals>.f at 0x7f7a0860cee0>
server = RedisServer(name='A', tcp_address=TCPAddress(host='localhost', port=55631), unixsocket='/tmp/aioredis.55631.sock', version=(5, 0, 3), password=None)

    @pytest.mark.skipif(sys.platform == 'win32',
                        reason="No unixsocket on Windows")
    async def test_connect_unixsocket_timeout(create_connection, server):
        with patch('aioredis.connection.open_unix_connection') as open_conn_mock:
            open_conn_mock.side_effect = lambda *a, **kw: asyncio.sleep(0.2)
            with pytest.raises(asyncio.TimeoutError):
>               await create_connection(server.unixsocket, db=0, timeout=0.1)

tests/connection_test.py:89:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/conftest.py:68: in f
    conn = await aioredis.create_connection(*args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = '/tmp/aioredis.55631.sock'

    async def create_connection(address, *, db=None, password=None, ssl=None,
                                encoding=None, parser=None, loop=None,
                                timeout=None, connection_cls=None):
        """Creates redis connection.

        Opens connection to Redis server specified by address argument.
        Address argument can be one of the following:
        * A tuple representing (host, port) pair for TCP connections;
        * A string representing either Redis URI or unix domain socket path.

        SSL argument is passed through to asyncio.create_connection.
        By default SSL/TLS is not used.

        By default any timeout is applied at the connection stage, however
        you can set a limitted time used trying to open a connection via
        the `timeout` Kw.

        Encoding argument can be used to decode byte-replies to strings.
        By default no decoding is done.

        Parser parameter can be used to pass custom Redis protocol parser class.
        By default hiredis.Reader is used (unless it is missing or platform
        is not CPython).

        Return value is RedisConnection instance or a connection_cls if it is
        given.

        This function is a coroutine.
        """
        assert isinstance(address, (tuple, list, str)), "tuple or str expected"
        if isinstance(address, str):
            address, options = parse_url(address)
            logger.debug("Parsed Redis URI %r", address)
            db = options.setdefault('db', db)
            password = options.setdefault('password', password)
            encoding = options.setdefault('encoding', encoding)
            timeout = options.setdefault('timeout', timeout)
            if 'ssl' in options:
                assert options['ssl'] or (not options['ssl'] and not ssl), (
                    "Conflicting ssl options are set", options['ssl'], ssl)
                ssl = ssl or options['ssl']

        if timeout is not None and timeout <= 0:
            raise ValueError("Timeout has to be None or a number greater than 0")

        if connection_cls:
            assert issubclass(connection_cls, AbcConnection),\
                    "connection_class does not meet the AbcConnection contract"
            cls = connection_cls
        else:
            cls = RedisConnection

        if loop is not None and sys.version_info >= (3, 8, 0):
            warnings.warn("The loop argument is deprecated",
                          DeprecationWarning)

        if isinstance(address, (list, tuple)):
            host, port = address
            logger.debug("Creating tcp connection to %r", address)
            reader, writer = await asyncio.wait_for(open_connection(
                host, port, limit=MAX_CHUNK_SIZE, ssl=ssl),
                timeout)
            sock = writer.transport.get_extra_info('socket')
            if sock is not None:
                sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
                address = sock.getpeername()
            address = tuple(address[:2])
        else:
            logger.debug("Creating unix connection to %r", address)
>           reader, writer = await asyncio.wait_for(open_unix_connection(
                address, ssl=ssl, limit=MAX_CHUNK_SIZE),
                timeout)
E           TypeError: cannot unpack non-iterable coroutine object

/opt/python38/lib/python3.8/site-packages/aioredis/connection.py:121: TypeError
______________________________________________________________________________________ test_create_connection_timeout[redis_v5.0.3] _______________________________________________________________________________________

create_pool = <function create_pool.<locals>.f at 0x7f7a087159d0>
server = RedisServer(name='A', tcp_address=TCPAddress(host='localhost', port=55631), unixsocket='/tmp/aioredis.55631.sock', version=(5, 0, 3), password=None)

    async def test_create_connection_timeout(create_pool, server):
        with patch('aioredis.connection.open_connection') as\
                open_conn_mock:
            open_conn_mock.side_effect = lambda *a, **kw: asyncio.sleep(0.2)
            with pytest.raises(asyncio.TimeoutError):
>               await create_pool(
                    server.tcp_address,
                    create_connection_timeout=0.1)

tests/pool_test.py:64:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/conftest.py:94: in f
    redis = await aioredis.create_pool(*args, **kw)
/opt/python38/lib/python3.8/site-packages/aioredis/pool.py:58: in create_pool
    await pool._fill_free(override_min=False)
/opt/python38/lib/python3.8/site-packages/aioredis/pool.py:383: in _fill_free
    conn = await self._create_new_connection(self._address)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = TCPAddress(host='localhost', port=55631)

    async def create_connection(address, *, db=None, password=None, ssl=None,
                                encoding=None, parser=None, loop=None,
                                timeout=None, connection_cls=None):
        """Creates redis connection.

        Opens connection to Redis server specified by address argument.
        Address argument can be one of the following:
        * A tuple representing (host, port) pair for TCP connections;
        * A string representing either Redis URI or unix domain socket path.

        SSL argument is passed through to asyncio.create_connection.
        By default SSL/TLS is not used.

        By default any timeout is applied at the connection stage, however
        you can set a limitted time used trying to open a connection via
        the `timeout` Kw.

        Encoding argument can be used to decode byte-replies to strings.
        By default no decoding is done.

        Parser parameter can be used to pass custom Redis protocol parser class.
        By default hiredis.Reader is used (unless it is missing or platform
        is not CPython).

        Return value is RedisConnection instance or a connection_cls if it is
        given.

        This function is a coroutine.
        """
        assert isinstance(address, (tuple, list, str)), "tuple or str expected"
        if isinstance(address, str):
            address, options = parse_url(address)
            logger.debug("Parsed Redis URI %r", address)
            db = options.setdefault('db', db)
            password = options.setdefault('password', password)
            encoding = options.setdefault('encoding', encoding)
            timeout = options.setdefault('timeout', timeout)
            if 'ssl' in options:
                assert options['ssl'] or (not options['ssl'] and not ssl), (
                    "Conflicting ssl options are set", options['ssl'], ssl)
                ssl = ssl or options['ssl']

        if timeout is not None and timeout <= 0:
            raise ValueError("Timeout has to be None or a number greater than 0")

        if connection_cls:
            assert issubclass(connection_cls, AbcConnection),\
                    "connection_class does not meet the AbcConnection contract"
            cls = connection_cls
        else:
            cls = RedisConnection

        if loop is not None and sys.version_info >= (3, 8, 0):
            warnings.warn("The loop argument is deprecated",
                          DeprecationWarning)

        if isinstance(address, (list, tuple)):
            host, port = address
            logger.debug("Creating tcp connection to %r", address)
>           reader, writer = await asyncio.wait_for(open_connection(
                host, port, limit=MAX_CHUNK_SIZE, ssl=ssl),
                timeout)
E           TypeError: cannot unpack non-iterable coroutine object

/opt/python38/lib/python3.8/site-packages/aioredis/connection.py:111: TypeError
___________________________________________________________________________________________ test_pool_idle_close[redis_v5.0.3] ____________________________________________________________________________________________

create_pool = <function create_pool.<locals>.f at 0x7f7a08365c10>, start_server = <function start_server.<locals>.maker at 0x7f7a0860caf0>, caplog = <_pytest.logging.LogCaptureFixture object at 0x7f7a073e6070>

    @pytest.mark.skipif(sys.platform == "win32", reason="flaky on windows")
    async def test_pool_idle_close(create_pool, start_server, caplog):
        server = start_server('idle')
        conn = await create_pool(server.tcp_address, minsize=2)
        ok = await conn.execute("config", "set", "timeout", 1)
        assert ok == b'OK'

        caplog.clear()
        with caplog.at_level('DEBUG', 'aioredis'):
            # wait for either disconnection logged or test timeout reached.
            while len(caplog.record_tuples) < 2:
                await asyncio.sleep(.5)
        expected = [
            ('aioredis', logging.DEBUG,
             'Connection has been closed by server, response: None'),
            ('aioredis', logging.DEBUG,
             'Connection has been closed by server, response: None'),
        ]
        if BPO_34638:
            expected += [
                ('asyncio', logging.ERROR,
                 'An open stream object is being garbage collected; '
                 'call "stream.close()" explicitly.'),
                ('asyncio', logging.ERROR,
                 'An open stream object is being garbage collected; '
                 'call "stream.close()" explicitly.')]
        # The order in which logs are collected differs each time.
>       assert sorted(caplog.record_tuples) == sorted(expected)
E       assert [('aioredis',...ponse: None')] == [('aioredis',...explicitly.')]
E         Right contains 2 more items, first extra item: ('asyncio', 40, 'An open stream object is being garbage collected; call "stream.close()" explicitly.')
E         Use -v to get the full diff

tests/pool_test.py:498: AssertionError
---------------------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------------------
DEBUG    aioredis:connection.py:205 Connection has been closed by server, response: None
DEBUG    aioredis:connection.py:205 Connection has been closed by server, response: None
==================================================================================================== warnings summary =====================================================================================================
tests/connection_test.py::test_connect_tcp_timeout[redis_v5.0.3]
tests/pool_test.py::test_create_connection_timeout[redis_v5.0.3]
  /opt/python38/lib/python3.8/site-packages/aioredis/connection.py:111: RuntimeWarning: coroutine 'sleep' was never awaited
    reader, writer = await asyncio.wait_for(open_connection(

tests/connection_test.py::test_connect_unixsocket_timeout[redis_v5.0.3]
  /opt/python38/lib/python3.8/site-packages/aioredis/connection.py:121: RuntimeWarning: coroutine 'sleep' was never awaited
    reader, writer = await asyncio.wait_for(open_unix_connection(

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================================================= short test summary info =================================================================================================
SKIPPED [1] tests/ssl_test.py:2: socat package required (apt-get install socat)
SKIPPED [2] tests/ssl_test.py:11: socat package required (apt-get install socat)
SKIPPED [1] tests/ssl_test.py:20: socat package required (apt-get install socat)
SKIPPED [1] tests/pubsub_receiver_test.py:237: json.loads accept bytes since Python 3.6
XFAIL tests/integration_test.py::test_operations[redis_v5.0.3-blocking_pop-1]
  blpop gets connection first and blocks
XFAIL tests/sentinel_failover_test.py::test_sentinel_slave_fail[redis_v5.0.3]
  Need proper sentinel configuration
XFAIL tests/sentinel_failover_test.py::test_sentinel_normal_fail[redis_v5.0.3]
  Need proper sentinel configuration
XFAIL tests/pyreader_test.py::test_maxbuf
XPASS tests/pool_test.py::test_select_and_create[redis_v5.0.3] Need to refactor this test
XPASS tests/sentinel_failover_test.py::test_sentinel_slave[redis_v5.0.3] same sentinel; single master;
FAILED tests/connection_test.py::test_connect_tcp_timeout[redis_v5.0.3] - TypeError: cannot unpack non-iterable coroutine object
FAILED tests/connection_test.py::test_connect_unixsocket_timeout[redis_v5.0.3] - TypeError: cannot unpack non-iterable coroutine object
FAILED tests/pool_test.py::test_create_connection_timeout[redis_v5.0.3] - TypeError: cannot unpack non-iterable coroutine object
FAILED tests/pool_test.py::test_pool_idle_close[redis_v5.0.3] - assert [('aioredis',...ponse: None')] == [('aioredis',...explicitly.')]
================================================================== 4 failed, 784 passed, 5 skipped, 4 xfailed, 2 xpassed, 3 warnings in 90.24s (0:01:30) ==================================================================
[root@localhost aioredis-1.3.1]#

[p5a0u9l]

Would really love to see a fix for this. I don't want to not upgrade nor do I want to refactor aioredis out of my library.

[honglei]

@p5a0u9l Try https://github.com/NoneGG/aredis

[p5a0u9l]

@honglei I did try aredis, and have now completely dropped aioredis. Unfortunately, running my code with 3.8 using aredis pubsub, there is an intermittent error. Same exact code, 3.7, no issues.

So, I'll stick with aredis, since it at least appears well supportes

[seandstewart]

This issue should be resolved as of #891 - please feel free to pull in the latest master and test to ensure this is the case.