opentelemetry collector -> APM server 401 error

[kobayashita]

Bug Report

What did you do?

I tried to connect opentelemetry collector -> APM server.

What did you expect to see?

I can connect and check the trace information on kibana.

What did you see instead? Under which circumstances?

cannot connect due to 401 error
"http.response.status_code":401,"error.message":"authentication failed"

Environment

• ECK version:

image version

• apm-server:8.13.0
• opentelemetry-collector-k8s:0.100.0

I use the following helm

- name: eck-stack
  version: 0.10.0 
  repository: "https://helm.elastic.co"

- name: opentelemetry-collector
  version: "0.91.0"
  repository: "https://open-telemetry.github.io/opentelemetry-helm-charts"

• Kubernetes information:

EKS

for each of them please give us the version you are using

k8s version

Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9-eks-036c24b

• Resource definition:

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server
  labels:
    helm.sh/chart: eck-apm-server-0.10.0
    app.kubernetes.io/name: eck-apm-server
    app.kubernetes.io/instance: cluster
    app.kubernetes.io/managed-by: Helm
  annotations:
    eck.k8s.elastic.co/license: basic
spec:
  version: 8.13.0
  count: 1
  config:
    logging.level: debug
  http:
    service:
      spec:
        ports:
        - name: http
          port: 8200
          targetPort: 8200
    tls:
      selfSignedCertificate:
        disabled: true
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: elk-ingress

spec:
  ingressClassName: nginx
  tls:
    - hosts:
      - example.com
      secretName: apm-tls-cert
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: apm-server-apm-http
            port:
              number: 8200
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: opentelemetry-collector
  namespace: cluster
  labels:
    helm.sh/chart: opentelemetry-collector-0.91.0
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/instance: cluster
    app.kubernetes.io/version: "0.100.0"
    app.kubernetes.io/managed-by: Helm
    
data:
  relay: |
    exporters:
      debug:
        verbosity: detailed
      otlphttp/elastic:
        endpoint: ${env:ELASTIC_APM_ENDPOINT}:443
        headers:
          Authorization: Bearer ${env:ELASTIC_APM_SECRET_TOKEN}
    extensions:
      health_check:
...

env

key	value
ELASTIC_APM_ENDPOINT	example.com
ELASTIC_APM_SECRET_TOKEN	MY_SECRET

• Logs:

{"log.level":"error","@timestamp":"2024-05-17T07:48:54.283Z","log.logger":"request","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1","file.name":"middleware/log_middleware.go","file.line":59},"message":"authentication failed","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector for Kubernetes/0.100.0 (linux/amd64)","source.address":"10.0.223.18","http.request.id":"a8950466-8c75-4b7a-84db-7a3cccfba2b2","event.duration":108680,"http.request.body.bytes":1356,"http.response.status_code":401,"error.message":"authentication failed","ecs.version":"1.6.0"}

I set according to the following document.

https://www.elastic.co/guide/en/observability/current/apm-open-telemetry-direct.html

APM settings are default except for the following

APM setting(Edit Elastic APM integration)

key	value
serverconfiguration.URL	example.com:443
API key for agent authentication	ON
Secret token	MY_SECRET