Change AccountService from Go to DotNet (auto)

[RassK]

Changes

Fixes #589

Changes AccountingService from Go to .NET service.
Uses OpenTelemetry .NET Automatic Instrumentation.

NOTE: It's a first preview of demo app that is instrumented with OpenTelemetry .NET Automatic instrumentation. Some parts are still TBD. Please review and add comments about improvements and missing parts / docs.

Merge Requirements

For new features contributions please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[RassK]

Example Jaeger traces:

CC: @lachmatt @Kielek Should parentless Kafka consume requests be filtered? Seems without additional context these just trash the output.

[RassK]

After removing EOF

[puckpuck]

@RassK Can you add a Chanlog entry for this? Also, we will need to update docs and the Helm chart for this. If you need help to prep a PR in those areas, please let us know.

[RassK]

@puckpuck thanks for helping to guide the last steps missing. I have updated the changelog. Would be glad to accept some help with other missing parts as well. It's my first PR to this repo.

[Kielek]

LGTM, I think this can be merged as the PR to opentlemetry.io is ready.
As I understand, helm charts should not be affected. We are changing just one type of service to another.

[puckpuck]

We will need to bump up the memory for the service in the Helm chart, but nothing beyond that. We can merge this before having to do anything in Helm, since Helm is bound to a release and not nightlies.

I want to keep the label on this PR so we can track all items that need to be updated when doing our next release.

[puckpuck]

When I run this branch, I notice clock skews with the accounting service, which causes issues with the trace waterfall. I'm not sure what this service does differently than our other .NET service (cart), but we don't see the same clock skew there. @RassK can you take a look at this?

[RassK]

@lachmatt could you recheck that kafka instrumentation, I think that is the source of this issue.

[RassK]

@puckpuck So consulted with @lachmatt and we reached conclusion that it is not the issue with clock skew.

In the .NET the Kafka span's instrumentation start time is the start of the polling. Previously we had issues with the amount of spans, so every poll (even an empty one) created a span. Now as we by default filter such spans, I could lower the polling max timeout to default (100ms), so accounting service's Kafka consume spans could be now <100ms, which can be still huge compared to some other services reporting processing time around 1ms. The default setting now renders the Jaeger UI in more understandable way... but we cannot make the UI look more beautiful just because of the messaging behaviour, it still has to reflect the actual logic.

I see it more like Jaeger could improve theirs's UI by using more dynamic time axle and collapse big portions like this (10s wait time).

Here is the improved look >

[julianocosta89]

@RassK wouldn't that be better handled in a linked span, instead of a child span?

We did that for fraudDetectionService.

[RassK]

@julianocosta89 that's unfortunately not configurable in auto instrumentation.

[puckpuck]

Something is wrong with the .NET SDK here. No other SDKs start the span when you start polling.

[lachmatt]

@puckpuck
This is a behavior of Kafka client instrumentation which is part of OTel .NET Autoinstrumentation.
Semantic conventions for messaging recommend to create receive spans for pull-based scenarios.
Most of the existing instrumentations for Kafka client are creating process spans, but these were problematic, as explained in OTEP, and in PR.

[puckpuck]

We should investigate doing this so the trace doesn't look like accounting service is starting before everything else, which looks like this should be done with a process consumer span. In its current state, this makes the consumption of the PlaceOrder trace very hard to do and will only raise questions from users about why the accounting service started before the request itself.

[lachmatt]

@RassK wouldn't that be better handled in a linked span, instead of a child span?

Creation context is used as a parent, in addition to being added as a link. This scenario is described in OTEP and was discussed when working on the instrumentation.

We should investigate doing this so the trace doesn't look like accounting service is starting before everything else, which looks like this should be done with a process consumer span. In its current state, this makes the consumption of the PlaceOrder trace very hard to do and will only raise questions from users about why the accounting service started before the request itself.

@pyohannes What would be the recommendation from Messaging SIG side in such scenario?