Add downstream dependency service name to logs and errors to improve alert insights

[sorenlouv]

Related: #183216

Changes

• Exclude APM error docs from logs and retrieve APM errors separately
• Get sample trace.id from logs and apm errors, and retrieve the downstream service name (if available)
• Minor prompt tweaks

Scenario

When running the Otel-Demo the "checkout" service is killed on purpose. This causes the failure rate of the frontend service to increase because is has a downstream dependency on the checkout service. This in turn causes alerts to be triggered.

When the user navigates to the alerts details page, and opens the insights they should be presented with the "checkout" service as the root cause.

Before

Before this change the alert insights did not capture that changes to the checkout service was the root cause

After

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[sorenlouv]

The downstream service name is resolved via a single sample trace id. The error could have multiple failed downstream dependencies. Ideally we'd get every downstream dependency for the given error. Not sure how in a performant manner

[sorenlouv]

buildkite test this

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[crespocarlos]

Code LGTM.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: cb57b71
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-183215-cb57b71e4919
• Observability Deployment

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
observability	286.0KB	286.3KB	+268.0B

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
module count	-	5407	+5407
total size	-	8.8MB	+8.8MB

History

• 💚 Build #209419 succeeded eab4222
• 💛 Build #209364 was flaky 4fe52c2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream