ddtrace/opentelemetry: refactor OTel API tests

[RomainMuller]

What does this PR do?

• Makes the mockTracerProvider return an empty JSON object on GET of URLs that aren't relevant to the test
• Uses an atomic pointer to hold the payload in the trace writer to remove a data race (this has been removed)
• The tests now marshal the trace payloads into a map that can be more easily accessed and more accurately tested, instead of checking exact string matches on the marshaled payloads.
• Removes timed contexts from all tests, and instead changes the function that waits for trace payloads to have a 10 second timer. This simplifies the code.

Motivation

This removes error logs (stemming from receiving an empty response) that otherwise can confuse someone trying to diagnose a bug into looking at the wrong problems.

This also addresses data corruption that happened if multiple concurrent accesses to the trace writer's payload happened (in particular in concurrency with flush).

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

For Datadog employees:

• If this PR touches code that handles credentials of any kind, such as Datadog API keys, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-01-25 17:36:08

Comparing candidate commit 8b46ec8 in PR branch romain.marcadier/diagnose-failure with baseline commit 882cddb in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 39 metrics, 2 unstable metrics.

[katiehockman]

This is ready for re-review. I don't think I should approve it, since I wrote a lot of the commits. @dianashevchenko can you take a look at this when you get the chance?
@RomainMuller I'm curious if you think d8f2dd5 is still worth having in this PR. If we do think it's worth including, we should probably try to create a test that can simulate the race that we're concerned about here. It's been removed.