Distributed Tracing - Option to propagate trace context when dispatching celery tasks

[msscaroso]

Is your feature request related to a problem? Please describe.

To illustrate, consider a process from a client calling a process in a server via a HTTP API, which then dispatches a task to do some background work (e.g. sending email, etc.).

sequenceDiagram
    Note over client: (A) Do stuff and log what happened
    client->>API:  POST /api (now: includes trace context)
    Note over API: (B) Do stuff and log what happened
    API->>queue: dispatch task (now: DOES NOT include trace context)
    API->>client: 200 Ok
    queue->>celery-worker: push
    Note over celery-worker: (C) Do stuff and log what happened

Right now, the client sends the trace context to the server and we are able to correlate what happened in A with B.
This is possible via the distributed tracing feature.

The celery-worker is instrumented by the agent, but the API process doesn't send the trace context to the queue and we can't correlate C with A/B.

Indeed, we might not want to correlate those when thinking about performance of the API (the API doesn't wait the worker to reply to the client. they are decoupled).

But, in the context that we are using the distributed tracing to correlate application logs of different process, it would be VERY useful to have the link between what's happening in the worker (C) with A and B.

Feature Description

Given that there are scenarios where we don't benefit from propagating the trace to the worker and other scenarios where we want this, I believe the agent could include the option to execute the context propagation to celery tasks.

To be sure, the very same diagram with the propagation in caps 👇

sequenceDiagram
    Note over client: (A) Do stuff and log what happened
    client->>API:  POST /api (now: includes trace context)
    Note over API: (B) Do stuff and log what happened
    API->>queue: dispatch task (after: CONTEXT PROPAGATION)
    API->>client: 200 Ok
    queue->>celery-worker: push
    Note over celery-worker: (C) Do stuff and log what happened

Describe Alternatives

Open telemetry client does that

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/__init__.py#L143-L154

https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumentation/opentelemetry-instrumentation-celery/src/opentelemetry/instrumentation/celery/utils.py#L135-L140

And datadog agent as well

https://ddtrace.readthedocs.io/en/stable/advanced_usage.html#examples

Additional context

If the idea is welcome, I'm willing to contribute ✋ , since the python agent is open source.

Thank you !!

Priority

[Really Want]

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Ak-x]

Hey @msscaroso; we already have a Feature Request for this; however since this is open-source you are more than welcome to make a contribution for our team to review.

cc: @ak-war

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.