Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

otel: simple OTEL collector/Prometheus stack for testing purposes #5026

Merged
merged 1 commit into from May 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
25 changes: 25 additions & 0 deletions hack/otel/README.md
@@ -0,0 +1,25 @@
# Sample stack for testing OTEL functionality with the CLI

To test the OTEL functionality present in the CLI, you can spin up a small demo compose stack that includes:
- an OTEL collector container;
- a Prometheus container;
- an Aspire Dashboard container

The `hack/otel` directory contains the compose file with the services configured, along with 2 basic configuration files: one for the OTEL collector and one for Prometheus.

## How can I use it?

1) Start the compose stack by running `docker compose up -d` in the `hack/otel/` directory;
2) Export the env var used to override the OTLP endpoint:
`export DOCKER_CLI_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317` (if running the CLI in a devcontainer or in other ways, you might have to change how you pass this env var);
3) Run the CLI to send some metrics to the endpoint;
4) Browse Prometheus at `http://localhost:9091/graph` or the Aspire Dashboard at `http://localhost:18888/metrics`;
5) In Prometheus, query `command_time_milliseconds_total` to see some metrics. In Aspire, select the resource in the dropdown.

> **Note**: The precise steps may vary based on how you're working on the codebase (buiding a binary and executing natively, running/debugging in a devcontainer, running the normal CLI as usual, etc... )

## Cleanup?

Run `docker compose down` in the `hack/otel/` directory.

You can also run `unset DOCKER_CLI_OTEL_EXPORTER_OTLP_ENDPOINT` to get rid of the OTLP override from your environment.
39 changes: 39 additions & 0 deletions hack/otel/compose.yaml
@@ -0,0 +1,39 @@
name: cli-otel

services:

prometheus:
image: prom/prometheus:latest
command:
- "--config.file=/etc/prometheus/prom.yaml"
ports:
# Publish the Prometheus frontend on localhost:9091
- 9091:9090
restart: always
volumes:
# Store Prometheus data in a volume:
- prom_data:/prometheus
# Mount the prom.yml config file
- ./prom.yaml:/etc/prometheus/prom.yaml
Comment on lines +16 to +17
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker for this PR, but I know that bind-mounting files can sometimes be problematic, so looking if we could have a directory to mount (which contains the config-file). Looks like /etc/prometheus already has some files in the image though, so if we do, we probably should use a different location for the config-file (but good news is that we already define it as a custom location above, so ... could still be an option);

docker run --rm --entrypoint /bin/sh prom/prometheus:latest -c 'ls -l /etc/prometheus'
total 4
lrwxrwxrwx    1 nobody   nobody          39 Apr 10 14:30 console_libraries -> /usr/share/prometheus/console_libraries
lrwxrwxrwx    1 nobody   nobody          31 Apr 10 14:30 consoles -> /usr/share/prometheus/consoles/
-rw-r--r--    1 nobody   nobody         934 Apr 10 14:21 prometheus.yml

The alternative could be to define it as a config: in the compose-file (but those are still implemented as bind-mounts in compose, so won't solve the "don't bind individual files" part)


aspire-dashboard:
image: mcr.microsoft.com/dotnet/nightly/aspire-dashboard:8.0-preview
ports:
- 18888:18888
environment:
DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS: 'true'

otelcol:
image: otel/opentelemetry-collector:latest
restart: always
depends_on:
- prometheus
- aspire-dashboard
ports:
- 4317:4317
volumes:
# Mount the otelcol.yml config file
- ./otelcol.yaml:/etc/otelcol/config.yaml
Comment on lines +34 to +36
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this one (also not a blocker)


volumes:
prom_data:
20 changes: 20 additions & 0 deletions hack/otel/otelcol.yaml
@@ -0,0 +1,20 @@
# Receive signals over gRPC and HTTP
receivers:
otlp:
protocols:
grpc:
http:

# Establish an endpoint for Prometheus to scrape from
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
otlp/aspire:
endpoint: aspire-dashboard:18889
tls::insecure: true

service:
pipelines:
metrics:
receivers: [otlp]
exporters: [prometheus, otlp/aspire]
6 changes: 6 additions & 0 deletions hack/otel/prom.yaml
@@ -0,0 +1,6 @@
# Configure Prometheus to scrape the OTel collector endpoint
scrape_configs:
- job_name: "otel-collector"
scrape_interval: 1s
static_configs:
- targets: ["otelcol:8889"]