dd-trace auto-instrumentation not capturing error and stack trace details for HTTP Exception errors in FastAPI

[sats268842]

Summary of problem
When using dd-trace auto-instrumentation for a Python web application, 500 errors are not showing the full error message and stack trace in Datadog. Instead, the traces just show "Missing error message and stack trace".

Which version of dd-trace-py are you using?
latest

Which version of pip are you using?
24.0

Which libraries and their versions are you using?
fastapi = "^0.110.0"

How can we reproduce your problem?
Set up a Fastapi app with dd-trace auto-instrumentation enabled
Trigger a 500 error by causing an exception (e.g. divide by zero)
Check Datadog traces and you'll see 500 errors but no error details

What is the result that you get?
Traces for 500 errors show "Missing error message and stack trace"

What is the result that you expected?
The traces should show the full error message and stack trace for the exception that caused the 500 error.

[emmettbutler]

Thanks for the report @sats268842. We'll see if we can put together a demo app that reproduces this behavior. If you have one you can share, that would be helpful.

[sats268842]

@emmettbutler Additionally, I've discovered that if you install both the auto instrumentation library injection and the ddtrace Python package, the tracing and metrics won't work. Also I will look into demo app, if time permits.

[sats268842]

@emmettbutler Also got workaround for this issue using below code

from ddtrace.opentelemetry import TracerProvider
from opentelemetry.trace import set_tracer_provider
set_tracer_provider(TracerProvider())


@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc):
    import traceback
    current_span = trace.get_current_span()
    if (current_span is not None) and (current_span.is_recording()):
        print(exc.__class__.__name__)
        exc_type, exc_value, exc_traceback = sys.exc_info()
        filename, line_number, function_name, code = traceback.extract_tb(exc_traceback)[-1]
        attributes = {
            "error.message": str(exc.detail),
            "error.kind": exc.__class__.__name__,
            "error.type": exc.__class__.__name__,
            "error.stack": traceback.format_exc(),
            "error.file": filename,
            "error.line": line_number,
            "error.function": function_name,
            "http.status_code": exc.status_code,
        }
        current_span.set_status(StatusCode.ERROR, str(exc.detail))
        current_span.set_attributes(attributes)

    return ORJSONResponse(({"detail": str(exc.detail)}), status_code=exc.status_code)

[quiqueg]

@sats268842 Do you have any middleware installed that are catching exceptions and not re-raising them? I was running into the same issue, and it was due to this middleware: https://gist.github.com/nymous/f138c7f06062b7c43c060bf03759c29e, which does something like:

response = Response(status_code=500)
try:
    response = await call_next(request)
except Exception:
    # do something
    raise
finally:
    # do something else
    return response

The problem was specifically with the return response inside of the finally block—this swallows the exception, so ddtrace's TracingMiddleware never receives the exception and is therefore never able to add it to the trace.

I fixed it by moving the return response to outside of the finally block, and being OK with not returning my own Response(status_code=500) from this middleware. I'm handling it in an exception handler instead:

@app.exception_handler(Exception)
async def handle_exceptions(exc: Exception):
    return Response(status_code=500)