[jaeger-v2] Fix e2e storage integration v0.99.0 OTEL col failing

[james-ryans]

Which problem is this PR solving?

• Fix failing CI at all of the v2 storage integration Bump the otel-collector group with 30 updates #5397

1. What was the observable problem after upgrade? I.e. how did it manifest?

My assumption was because of OTEL col changed the ToClientConn gRPC implementation from DialContext to NewClient here open-telemetry/opentelemetry-collector#9965. Both integration spanWriter and jaegerexporter use OTLP exporter which creates the gRPC connection through the ToClientConn above.

2. What do you believe the root cause was?

The gRPC connection from OTLP exporter has not yet been successfully settled but the requests have been sent.

From the #5397 PR failing CI at Badger and gRPC storages, Badger failed only at GetServices and GetOperations while gRPC failed at GetServices, GetOperations, and FindTraces.

Since both of them failed at GetServices and GetOperations which is the beginning of the tests, the "spanWriter OTLP exporter" to "OTEL col binary OTLP receiver" connection probably has not yet been successfully settled.

Then, the Badger storage doesn't have any issue with the FindTraces because it cleanup the storage with purge() functionality but the gRPC restarts the remote storage server. With the same assumption, the gRPC connection from OTEL col binary to the remote storage server probably has not been settled yet.

3. How does the change fix the root cause?

Retrying the requests solves the issue. Here are my arguments:

• The spanWriter's OTLP exporter retries only at GetServices after the retry config has been enabled, previously requests failed at GetServices and GetOperations.
• The jaegerexporter's OTLP exporter has the same error message "plugin error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:17271: connect: connection refused" but success at the 2nd attempt.

Description of the changes

• Change the spanWriter log level to DEBUG
• Enable retry to integration spanWriter

How was this change tested?

• Run make jaeger-v2-storage-integration-test until it fails since it has a low probability of reproducing the same error

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.59%. Comparing base (13cbaed) to head (24fc3e0).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5419      +/-   ##
==========================================
+ Coverage   94.54%   94.59%   +0.04%     
==========================================
  Files         346      346              
  Lines       16947    16953       +6     
==========================================
+ Hits        16022    16036      +14     
+ Misses        724      716       -8     
  Partials      201      201              

Flag	Coverage Δ
badger_v1	10.28% <ø> (-0.01%)	⬇️
badger_v2	6.36% <100.00%> (-0.21%)	⬇️
cassandra-3.x	?
cassandra-3.x-v1	18.12% <ø> (?)
cassandra-3.x-v2	6.13% <100.00%> (?)
cassandra-4.x	?
cassandra-4.x-v1	18.12% <ø> (?)
cassandra-4.x-v2	6.13% <100.00%> (?)
elasticsearch-5.x	5.74% <100.00%> (-0.17%)	⬇️
elasticsearch-6.x	5.74% <100.00%> (-0.16%)	⬇️
elasticsearch-7.x	5.74% <100.00%> (-0.16%)	⬇️
elasticsearch-8.x	5.74% <100.00%> (-0.16%)	⬇️
grpc_v1	12.61% <ø> (-0.02%)	⬇️
grpc_v2	11.35% <100.00%> (-0.19%)	⬇️
kafka	9.95% <ø> (-0.01%)	⬇️
opensearch-1.x	5.74% <100.00%> (?)
opensearch-2.x	5.74% <100.00%> (-0.16%)	⬇️
unittests	91.41% <50.00%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yurishkuro]

Is this related to #5418?

[james-ryans]

Is this related to #5418?

No, I haven't figured out why cassandra v2 test is failing.

[yurishkuro]

Try to add more details to the description.

1. What was the observable problem after upgrade? I.e. how did it manifest?
2. What do you believe the root cause was?
3. How does the change fix the root cause?

[james-ryans]

Do you think adding retry to integration spanWriter and jaegerexporter is good enough? Actually, I wasn't able to break down the cause of this issue since it is hard to replicate locally.

Try to add more details to the description.

I've added the explanation to this PR description