Support Python 3.11

[Yun-Kim]

This issue is to track our progress in supporting Python 3.11.

Update (April 3, 2023): The ddtrace library now supports Python 3.11, with the exception of certain integrations for which libraries are not themselves compatible with Python 3.11. This includes celery, which should be resolved once they release version 5.3.

The dd-trace-py library currently does not support Python 3.11, which is scheduled to be released October 24 2022. We have several integrations and core components that are failing to build or failing tests with #4125.

Checklist for dd-trace-py core components:

1. ✅ ddtrace-run: ~gevent dependency fails to install due to API changes to their vendored greenlet. ~

• Waiting on gevent to unblock and support 3.11 issue and PR.
• Updated test suite to run with Python 3.11 and gevent:latest.

2. ✅ debugger (Dynamic Instrumentation), internal: depend on the bytecode module which doesn't support Python 3.11 yet, so the debugger and internal test suites won't pass.

• skipping Python 3.11 tests until bytecode supports Python 3.11.
• Bytecode now supports 3.11, but our injection and wrapping bytecode needs to be updated to match Python 3.11's new bytecode op codes.
• Dynamic instrumentation will support Python 3.11 once feat: add Python 3.11 support to Dynamic Instrumentation #4854 is merged.

3. ✅ profiler: see feat(profiler): add Python 3.11 support #4343 description, the profiler is still crashing on Python 3.11. The profiler now supports Python 3.11 as of feat(profiling): add support for Python 3.11 #4511.

• There are some known compatibility issues with the profiler on Python 3.11. See fix(profiler): indirectly access frame objects for tracebacks for Python 3.11 #4895.
• Compatibility issues should be resolved with fix(profiler): handle non-frame objects while stack-unwinding #5019

4. ✅ tracer: Riot seems to have problems with Python 3.11's importlib in terms of managing virtual environments. Specifically, Hypothesis' vendored text file is not being found even though there is no problem running the test locally with Python 3.11.

• Workaround was to ignore the failing test file that imported from hypothesis.provisional, and create a separate test suite using tox to test it separately.

Checklist for integrations:

1. ❌ celery: getting a AttributeError: '_Code' object has no attribute 'co_positions' error for 2 celery tests, failing tests marked as flaky but they are constantly failing with PR feat(tracer): add Python 3.11 support #4125

• Skipping testing with Python 3.11 for now as celery's dependency billiard is not yet Python 3.11 compatible. See issue
• Waiting for celery to release a Python 3.11 compatible version, should be released in celery=5.3.0

2. ✅ cherrypy & pynamodb: inspect.getargspec() has been deprecated since Python 3.0 and was now removed in Python 3.11.

• Known issue and resolved by cherrypy fix, released in version 18.7.
• In other words Python 3.11 is only compatible with Cherrypy >= 18.7.
• Pynamodb released compatibility fix for removed getargspec() in version 4.1
• Updated cherrypy & pynamodb test suites to use versions 18.7 & 4.1 respectively with Python 3.11.

3. ✅ dogpilecache: inspect.getargspec() has been deprecated since Python 3.0 and was now removed in Python 3.11.

• Known issue and resolved in dogpilecache >= 0.7
• Updated dogpilecache test suite to run with Py 3.11 with dogpilecache >= 0.7.

6. ✅ kombu: failing in CI due to a pinned dependency.

• Kombu's dependency Vine==1.3.0 is pinned in CI for some reason and it is incompatible with Python 3.11 (inspect.formatargspec() has been deprecated since Python 3.5 and was now removed in Python 3.11.)
• kombu >= 5.0 is compatible with Python 3.11, updated kombu test suite to run with kombu >= 5.0

7. ✅ django: Python's gettext.py module removed a deprecated function argument codeset as of Python 3.11.

• Django 4.1 is meant to be first django version compatible with Python 3.11
• Updated django test suite to run Py3.11 with Django 4.1.

8. ✅ fastapi & starlette:

• Getting TypeError: Passing coroutines is forbidden, use tasks explicitly. errors from asyncio
• fastapi giving the same error, makes sense as it is based on and dependent on starlette
• starlette released Python 3.11 compatible version as of v0.21.0, updated starlette and fastapi test suites to use starlette >= 0.21.0 with Python 3.11.

9. ✅ graphene, graphql:

• Getting error as they are also dependent on the bytecode module, which doesn't yet support Python 3.11
• skipping Python 3.11 tests until bytecode supports Python 3.11.
• Bytecode now supports 3.11, but our injection and wrapping bytecode needs to be updated to match Python 3.11's new bytecode op codes.
• Will be unblocked once feat: add Python 3.11 support to Dynamic Instrumentation #4854 is merged.

10. ✅ grpc:

• According to grpc maintainers grpc 1.49.0 will support python 3.11 which should be released soon.
• Updated grpc test suites to use 1.49.0 with Python 3.11.
• Note: grpcio with pytest-asyncio is causing Python segmentation faults on the test_unary_exception and test_unary_cancellation test cases. You can reproduce this using this gist. Thus I am skipping Python 3.11 tests with grpcio and pytest-asyncio.

11. ✅ mariabd: fails to install with Python 3.11 currently, version 1.1.2 onwards should be compatible with Python 3.11.

• Updated mariadb test suite to use 1.1.2+ for Python 3.11.

12. ✅ asyncpg: ~getting asyncpg/pgproto/pgproto.c:223:12: fatal error: longintrepr.h: No such file or directory #include "longintrepr.h" error during install process

• Should be fixed in coming release of asyncpg. See issue~
• Skipping the asyncpg python 3.11 tests for now until asyncpg releases a compatible version.
• Updated asyncpg test suite to use latest (0.27.0) with Python 3.11.

13. ✅ aioredis: failing to run test cases with TypeError: duplicate base class "TimeoutError"

• According to this aioredis issue, aioredis is being replaced by the redis library (4.2.0+), and no longer supports Python 3.11 (redis will take over support for aioredis). Therefore, the aioredis test suite has been update to exclude Python 3.11 and an internal issue has been raised to transition our integration/testing for aioredis into our redis integration.

14. ✅ psycopg: failing to install.

• According to psycopg docs, version 2.9.2 has preliminary support for python 3.11, but our riotfile tests are based on ~=2..8.0, ~=2.9.0, latest
• Updated psycopg test suite to use 2.9.2+ with Python 3.11.

15. ✅ pylibmc: fails to build with Python 3.11, fixed in pylibmc==1.6.1

• Updated pylibmc test suite to use 1.6.1+ with Python 3.11.

16. ✅ gevent: fails to build with Python 3.11.

• gevent uses a vendored version of greenlet, which is not yet compatible with Python 3.11 (See issue)
• Skipping the gevent python 3.11 tests for now until gevent releases a compatible version.
• Updated gevent test suite to use 22.8+ with Python 3.11.

[Yun-Kim]

Currently running into a problem with gevent failing to build properly with Python 3.11. I suspect it has to do with gevent's vendored version of greenlet, and I've made an issue on gevent's github page here.

[Yun-Kim]

Note that I've separated the profiler changes into a separate PR #4343 so that it can unblock the tracer PR #4125.

[fukszssl247]

docker python just updated to 3.11 wheel fails to build if used with python 3 latest

[Yun-Kim]

Status Update: The tracer has preliminary support for Python 3.11 as of version 1.6.0. This unfortunately does not yet include integrations for asyncpg, graphene, graphql, celery, and grpcio with pytest-asyncio, as well as the profiler and dynamic instrumentation.

[atrigent]

FYI, the new version of bytecode supporting 3.11 was just released: https://github.com/MatthieuDartiailh/bytecode/blob/main/doc/changelog.rst#2022-11-30-version-0140

A version of asyncpg with prebuild wheels has also been released.

[bryanhelmig]

We're running into this as well. Would love a fix!

[Yun-Kim]

Hi @bryanhelmig, could you clarify what problem you're running into?
As of right now, the ddtrace library supports tracing and most instrumented libraries on Python 3.11, with profiling and dynamic instrumentation not yet supported. Thanks!

[eqperes]

We're currently block by the support of pytest-asyncio with the following error raised on our tests:

AttributeError: module 'asyncio.coroutines' has no attribute '_DEBUG'

[Yun-Kim]

Hi @eqperes, would you mind opening up a new issue so we can look into it in more detail? Adding a more detailed traceback/error log would be helpful. Thanks!

[BobACollins]

I'm running several of the not yet supported integrations. I'm happy to create a bug report with what errors I am getting or post here if that would help.

[Yun-Kim]

@BobACollins thanks for reaching out, please do open up a new issue with your bug report!

[AlonMorgen]

So, just to make it clear - what is not supported yet in python3.11?
Does "dynamic instrumentation" meaning manual instrumentation of the application?
Or only the code profiling?

[P403n1x87]

@AlonMorgen we've had reports of persisting issues when the profiler is on with Python 3.11. We have fixed a couple of those that were caused by the profiler directly. There might still be issues when using the profiler with CPython 3.11 which might be indirectly caused by dd-trace-py. Because of the number of issues related how CPython 3.11 handles frame objects that have been fixed recently, we suspect that in some cases our profiler is triggering unexpected behaviour in CPython. Our advice would be to ensure that you are always running with the latest release of CPython 3.11 to ensure that you have those fixes in your environments as they come out. If you notice unexpected behaviour while using the profiler with the latest release of CPython 3.11, our advice is to temporarily turn it off, and open a new issue if one does not exist already that matches the problem you are witnessing.

[Yun-Kim]

@AlonMorgen other than profiling as @P403n1x87 mentioned above, all core ddtrace components (i.e. Tracing, Dynamic Instrumentation) support 3.11, along with almost all tracing integrations (with the exception of celery, which itself does not yet support Python 3.11).

Of course, if you run into errors while running any of our integrations with Python 3.11, please open a support issue so we can resolve it.

[ayyoubelamrani4]

Hi there

After upgrading to python 3.11 and ddtrace 1.8 (from 1.6). We stopped receiving the graphql.request traces, this is very critical for our business.

Not sure if it's related but I found this in the logs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/site-packages/ddtrace/internal/periodic.py", line 73, in run
    self._target()
  File "/usr/local/lib/python3.11/site-packages/ddtrace/profiling/collector/__init__.py", line 42, in periodic
    for events in self.collect(): 

and

Exception in thread ddtrace.profiling.collector.memalloc:MemoryCollector:

A screenshot showing that we stopped receiving traced for graphql.request:

Is it something planned to be fixed soon or should we roll back to python 3.10 ...?

[ayyoubelamrani4]

Is ddtrace 1.9 fixing this? Any eta for the release?

[Yun-Kim]

Hi @ayyoubelamrani4, the PR to add support for graphql/graphene in Python 3.11 has not yet been released, and should be in 1.9. In the meantime, I'd suggest roll backing to Python 3.10 until 1.9 is released.

The goal is to get 1.9 released officially by EOW next week or early the week after. If the issue still persists after upgrading, please open a new support issue.

[lanprijatelj]

Hi @Yun-Kim I'm just upgrading our python version to 3.11.2 and I'm facing the following error with dd-trace:

File "/usr/local/lib/python3.11/site-packages/ddtrace/internal/module.py", line 216, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.11/site-packages/ddtrace/contrib/aiopg/__init__.py", line 25, in <module>
    from .patch import patch
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/usr/local/lib/python3.11/site-packages/ddtrace/internal/module.py", line 216, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.11/site-packages/ddtrace/contrib/aiopg/patch.py", line 7, in <module>
    from ddtrace.contrib.aiopg.connection import AIOTracedConnection
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "/usr/local/lib/python3.11/site-packages/ddtrace/internal/module.py", line 216, in _exec_module
    self.loader.exec_module(module)
  File "/usr/local/lib/python3.11/site-packages/ddtrace/contrib/aiopg/connection.py", line 22, in <module>
    class AIOTracedCursor(wrapt.ObjectProxy):
  File "/usr/local/lib/python3.11/site-packages/ddtrace/contrib/aiopg/connection.py", line 30, in AIOTracedCursor
    @asyncio.coroutine
     ^^^^^^^^^^^^^^^^^
AttributeError: module 'asyncio' has no attribute 'coroutine'

After checking the codebase I can still see the @asyncio.coroutine decorator being used instead of async def as instructed in: Official Python documentation
Can we hope to get a fix for this issue?

[Yun-Kim]

Hi @lanprijatelj, thanks for reaching out.
It looks like aiopg==1.4.0 is the first known version to support Python 3.11, but we don't currently support aiopg>=1.0. I suspect the asyncio.coroutine syntax error is not the only thing blocking our support for aiopg==1.4.0 and Python 3.11. In the meantime, I would recommend you to avoid upgrading to Python 3.11, or feel free to open an issue for a feature request to support aiopg>=1.0.

[brentleeper]

Is python 3.11 supported or not?

The documentation suggests that it is here: https://ddtrace.readthedocs.io/en/stable/release_notes.html#v1-9-0 -> dynamic instrumentation: Python 3.11 is now supported.

and here: https://docs.datadoghq.com/tracing/trace_collection/compatibility/python/ -> RUNTIME VERSION - 2.7, 3.5-3.11

But this issue is still opened, which suggests that it is not.

If 3.11 is supported, close this issue. If it’s not, correct your documentation.

[fredrikaverpil]

@brentleeper we use 3.11 and we use datadog too. I guess you have to check which libraries you need support for (top post in this thread).

[brentleeper]

Thank you @fredrikaverpil.

@DataDog If there is only partial support, update your documentation to include a notice since this could mislead customers into thinking that 3.11 is FULLY supported.

[Yun-Kim]

Hi @brentleeper, sorry for the confusion. The core components checklist in the top post in this thread indicate that ddtrace fully supports Python 3.11. However, certain integrations (most notably celery) are not yet compatible with Python 3.11, which is outside our control.

Since this issue was meant to track the core components of ddtrace, which fully supports Python 3.11, and it's out of our control when celery will be releasing a Python 3.11-compatible version (we expected earlier in 2022 but alas), I'll be closing this issue to avoid further confusion.

For any 3.11 support issues in the future, please create a new issue.

[danyi1212]

❌ celery: getting a AttributeError: '_Code' object has no attribute 'co_positions' error for 2 celery tests, failing tests marked as flaky but they are constantly failing with PR #4125

• Skipping testing with Python 3.11 for now as celery's dependency billiard is not yet Python 3.11 compatible. See Python 3.11: '_Code' object has no attribute 'co_positions' celery/billiard#377
• Waiting for celery to release a Python 3.11 compatible version, should be released in celery=5.3.0

Celery 5.3.0 is released now, and Celery supports Python 3.11.
Any plans to support it and fix the tests? Thanks!

@Yun-Kim

[Yun-Kim]

Hi @danyi1212, could you create a separate issue for supporting celery? This issue is closed and will be difficult to respond to support requests through this issue. Thank you!

[wlach]

Hi @lanprijatelj, thanks for reaching out. It looks like aiopg==1.4.0 is the first known version to support Python 3.11, but we don't currently support aiopg>=1.0. I suspect the asyncio.coroutine syntax error is not the only thing blocking our support for aiopg==1.4.0 and Python 3.11. In the meantime, I would recommend you to avoid upgrading to Python 3.11, or feel free to open an issue for a feature request to support aiopg>=1.0.

FWIW I filed #8229 for this