Interesting; the list of headers to exclude from signing in JSv2:

Maybe this issue belongs in the cross-SDK repo?

Thanks @benkehoe

It is surprising that they differ significantly. I think for now this issue only crops up with the js client because the javascript https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-instrumentation-http adds the traceparent header to every request.

Adding an AWS specific code path to the HTTP instrumentation felt weird so I raised this here

I'm confused. Is the OpenTelemetry JavaScript implementation doing something that is not standard for OpenTelemetry? If so, it seems like the OpenTelemetry implementation should be changed. If the OpenTelemetry JavaScript implementation is following the OpenTelemetry spec, we should expect that, for example, a Python implementation of OpenTelemetry would also cause signing issues for the Python SDK (boto3)

I'm always confused 😆

The reason I opened this here is that I think the unexpected behaviour is coming from the aws-sdk rather than the instrumentation-http package.

The instrumentation-http package passes the traceparent to all downstream HTTP endpoints. There are conversations about being able to allow/deny propagation for specific endpoints open-telemetry/opentelemetry-js#1698 but I don't think anyone is actively working on this. This would still lead to friction for new open telemetry users wanting to instrument aws-sdk requests.

When the request retries at the moment the signature is not recalculated for subsequent HTTP requests even though some of the headers are now different. I think this is because of the signature caching logic. Does boto cache the signature? I don't see this in the aws-sdk-js v3 which might be why we don't see the problem there

edit:

@benkehoe I think boto3 doesn't cache the token so I would guess when the request is retried a new signature is created
https://github.com/boto/botocore/blob/1fad151309322a88641571ac188af8e68ba56a59/botocore/auth.py#L444C23-L444C23

ah, a signature caching issue makes sense

Hi @Ankcorn,

Thanks for raising the issue. Aside from botocore did you notice this issue with other SDKs?
Newer SDKs have a more uniform list of unsignable headers as they all should follow the same type of internal RFC like paper. Older SDKs implementations might differ.

While we look into adding the traceparent header into the the list, you can workaround it by doing:

AWS.Signers.V4.prototype.unsignableHeaders.push('traceparent');

Thanks,
Ran~

Hey @RanVaknin

Thanks for the workaround, that helps a lot :) I have confirmed that it works.

I have taken a good look at botocore and the aws-sdk-js v3.

It helped me realise here that the root cause for the js v2 issue was not the headers but the caching of the signature, which we only noticed because of the traceheader changinf. The js v2 SDK seems to be the only one that does this.

Hope that update helps :)

This PR seems to fix the issue: open-telemetry/opentelemetry-js#4346
which is released in @opentelemetry/instrumentation-http v0.46.0: open-telemetry/opentelemetry-js#4358