This tutorial step covers the basic usage of the OpenTelemetry Collector and OpenTelemetry Operator.
By today the OpenTelemetry Operator offers two CustomResouceDefinitions.
-
The
v1alpha1.Instrumentationcan be used to configure applications that are configured with the OpenTelemetry-SDK and injection of auto-instrumentation libraries. Currently Apache HTTPD, DotNet, Go, Java, Nginx, NodeJS and Python are supported. Readme -
The
v1alpha1.OpenTelemetryCollectorsimplifies the operation of the OpenTelemetry Collector on Kubernetes. There are different deployment modes available, breaking config changes are migrated automatically, provides integration with Prometheus (including operating on Prometheus Operator CRs) and simplifies sidecar injection.
Make sure the demo application is deployed and running:
kubectl apply -f https://raw.githubusercontent.com/pavolloffay/kubecon-na-2023-opentelemetry-kubernetes-metrics-tutorial/main/app/k8s.yamlTo create an Instrumentation resource for our sample application run the following command:
kubectl apply -f https://raw.githubusercontent.com/pavolloffay/kubecon-na-2023-opentelemetry-kubernetes-metrics-tutorial/main/backend/04-metrics-auto-instrumentation.yamlUntil now we only have created the Instrumentation resource, in a next step you need to opt-in your services for auto-instrumentation. This is done by updating your service's spec.template.metadata.annotations.
Before applying the annotation let's take a look at the pod specification:
kubectl get pods -n tutorial-application -l app=backend2 -o yamlAll you need to do now, is to inject the configuration:
kubectl patch deployment backend2-deployment -n tutorial-application -p '{"spec": {"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"true"}}}} }'In case of using the manual instrumentation, the follwoing annotation would be set to only configure the SDK:
instrumentation.opentelemetry.io/inject-sdk: "true"Now verify that it worked:
kubectl get pods -n tutorial-application -l app=backend2 -o yamland access metrics (kubectl port-forward -n observability-backend svc/prometheus 8080:80)
or traces (kubectl port-forward -n observability-backend svc/jaeger-query 16686:16686).
Example how to configure Node.JS.
The OpenTelemetry Collector can be divided into a few major components.
- Receivers: Collect data from a specific source, like an application or infrastructure, and convert it into pData (pipeline data). This component can be active (e.g. Prometheus) or passive (OTLP).
- Processors: Manipulates the data collected by receivers in some way. For example, a processor might filter out irrelevant data, or add metadata to help with analysis. Examples include the batch or metric renaming processor.
- Exporters: Send data to an external system for storage or analysis. Examples of exporters are Prometheus, Loki or the OTLP exporter.
- Extensions: Add additional functionality to OpenTelemetry collector that is not strictly related to the telemetry data, like configuring a bearer token or offering a Jaeger remote sampling endpoint.
- Connectors: Is both an exporter and receiver. It consumes data as an exporter in one pipeline and emits data as a receiver in another pipeline.
The components are composed into pipelines, that are separated by signal type (metrics, traces, logs). Each datapoint is then goes through the chain consisting receiver(s) -> processor(s) -> exporter(s). For more details, check the offical documentation.
The configuration of the Open Telemetry Collector is described in YAML. The following shows an OTLP/gRPC receiver listening on localhost:4317. A batch processor with default parameters and a logging exporter with a normal log level. It also describes multiple pipelines for different telemetry data, which all route their collected telemetry data to the logging exporter.
The easiest way to learn more about the configuration options of individual components is to visit the readme in the component folder directly. Example debugexporter.
receivers:
otlp:
protocols:
grpc:
endpoint: 127.0.0.1:4317
processors:
batch:
exporters:
debug:
verbosity: detailed
service:
pipelines:
metrics:
receivers: [otlp]
processors: [batch]
exporters: [debug]In previous section, we have seen how to instrument your application both manually and automatically, with the help of the Instrumentation custom resource. In this section, we will take it one step further and introduce the actual OpenTelemetry Collector to the equation. Instead of sending the telemetry signals directly into our backend, we will send them to a collector that will be created by the operator, based on our CR definition.
First, lets inspect our OpenTelemetryCollector CR we have prepared. We will use the the basic configuration we have introduced previously and make the collector export the application metrics to the standard output, with the help of the debug exporter.
Apart from the config and other commonly found parts of the spec you might see in other Kubernetes resources (such as image, replicas, resources or even autoscaler), there are couple of parameters that are specific to the OpenTelemetry operator logic:
mode- Defines the deployment mode of the collector. This needs to one of the following values:deployment,daemonset,statefulset,sidecar. The default value isdeployment.upgradeStrategy- Decides whether the collector instance should be automatically updated by the operator or not. Possible values areautomatic,none. The default value isautomatic.targetAllocator- Determines whether an instance of a target allocator should be created. Target allocator is a special, optional component that decouples service discovery and metric collection functions of Prometheus in a way that allows for their independent scaling. It will be introduced in more details in section 5. By default, target allocator is not created.serviceAccount- Service account to be used by the collector. Important especially for collector configured with receivers that are responsible for collecting telemetry from the Kubernetes cluster, which requires the collector to have appropriate permissions.
Finally, let's create our OpenTelemetryCollector CR:
kubectl apply -f https://raw.githubusercontent.com/pavolloffay/kubecon-na-2023-opentelemetry-kubernetes-metrics-tutorial/main/backend/04-collector-basic.yamlLet's verify our CR was created correctly:
kubectl get -n observability-backend opentelemetrycollectors.opentelemetry.ioExpected output:
NAME MODE VERSION READY AGE IMAGE MANAGEMENT
otel-basic deployment 0.88.0 1/1 5m55s ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.88.0 managedAnd see that the collector is actually running:
kubectl get -n observability-backend -l app.kubernetes.io/managed-by=opentelemetry-operator podExpected output:
NAME READY STATUS RESTARTS AGE
otel-basic-collector-5bf5fbd9d9-prqtb 1/1 Running 0 6m39sAt last, let's check the logs of the collector to see that it is actually collecting the metrics from our application:
kubectl logs -n observability-backend deployments/otel-basic-collectorWait a minute, I don't see any metrics being printed out. What's missing? That's right, we still haven't updated our Instrumentation CR to point it to our new collector instance. Let's do that now by applying updated Instrumentation CR with the following command:
kubectl apply -f https://raw.githubusercontent.com/pavolloffay/kubecon-na-2023-opentelemetry-kubernetes-metrics-tutorial/main/backend/04-metrics-auto-instrumentation-collector.yamlCheck the logs again and observe the metrics being printed. That looks better!
Now we understand and we've seen how the Instrumentation and OpenTelemetryCollector CRs work independently and together. In next sections, we'll dive deeper into the configuration of the collector and see how we can use the collector and operator CRs to collect metrics from our Kubernetes cluster.