Istio-gateway not generate spanId, traceId, parentSpanId #50911

patsevanton · 2024-05-08T05:06:22Z

patsevanton
May 8, 2024

Hello! Thanks for Istio!
I try enable generate spanId, traceId, parentSpanId by Istio-gateway.
But spanId=null, traceId=null, parentSpanId=null
How generate spanId, traceId, parentSpanId for Istio-gateway?

version

helm list -n istio-system
NAME          NAMESPACE     REVISION  UPDATED                                 STATUS    CHART               APP VERSION
istio-base    istio-system  1         2024-03-14 16:10:52.74343861 +0000 UTC  deployed  base-1.20.6         1.20.6   
istio-istiod  istio-system  32        2024-05-08 05:10:02.604411197 +0000 UTC deployed  istiod-1.20.6       1.20.6 

helm list -n istio-ingress
NAME          NAMESPACE     REVISION  UPDATED                                 STATUS    CHART           APP VERSION
istio-gateway istio-ingress 2         2024-03-26 08:22:33.462839072 +0000 UTC deployed  gateway-1.20.6  1.20.6

Config:

helm get values -n istio-system istio-istiod
USER-SUPPLIED VALUES:
global:
  proxy:
    tracer: none
meshConfig:
  accessLogEncoding: JSON
  accessLogFile: /dev/stdout
  accessLogFormat: |
    {
      "traceId": "%REQ(x-b3-traceid)%",
      "spanId": "%REQ(x-b3-spanid)%",
      "parentSpanId": "%REQ(x-b3-parentspanid)%"
    }
  defaultConfig:
    holdApplicationUntilProxyStarts: true
    tracing: {}
  defaultProviders:
    tracing:
    - tempo
  enableTracing: true
  extensionProviders:
  - envoyExtAuthzHttp:
      headersToDownstreamOnDeny:
      - content-type
      - set-cookie
      headersToUpstreamOnAllow:
      - authorization
      - cookie
      - path
      - x-auth-request-user
      - x-auth-request-email
      - x-auth-request-access-token
      - x-forwarded-access-token
      includeRequestHeadersInCheck:
      - authorization
      - cookie
      - x-forwarded-access-token
      port: "80"
      service: admin-mycompany-oauth2-proxy.admin-mycompany.svc.cluster.local
      timeout: 1.5s
    name: admin-mycompany-oauth2-proxy
  - envoyExtAuthzHttp:
      headersToDownstreamOnDeny:
      - content-type
      - set-cookie
      headersToUpstreamOnAllow:
      - authorization
      - cookie
      - path
      - x-auth-request-user
      - x-auth-request-email
      - x-auth-request-access-token
      - x-forwarded-access-token
      includeRequestHeadersInCheck:
      - authorization
      - cookie
      - x-forwarded-access-token
      port: "80"
      service: extra-value-admin-oauth2-proxy.extra-value-admin.svc.cluster.local
      timeout: 1.5s
    name: extra-value-admin-oauth2-proxy
  - name: tempo
    opentelemetry:
      port: 4317
      service: opentelemetry-collector.opentelemetry-collector.svc.cluster.local
pilot:
  autoscaleMin: 2

Telemetry tracing providers

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: tempo
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: tempo
      randomSamplingPercentage: 100

logs istio-gateway:

stern -n istio-ingress . | grep spanId
istio-gateway-dnsgx istio-proxy {"parentSpanId":null,"spanId":null,"traceId":null}
istio-gateway-7mtbv istio-proxy {"traceId":null,"parentSpanId":null,"spanId":null}

helm get values -n opentelemetry-collector opentelemetry-collector

USER-SUPPLIED VALUES:
config:
  exporters:
    logging:
      verbosity: detailed
    otlp/tempo:
      endpoint: tempo-opentelemetry.mycompany.corp:4317
      retry_on_failure:
        enabled: true
      sending_queue:
        enabled: true
      tls:
        insecure: true
  processors:
    batch:
      send_batch_max_size: 3072
      send_batch_size: 3072
      timeout: 200ms
    resource:
      attributes:
      - action: upsert
        key: cluster
        value: dev
      - action: upsert
        key: domain
        value: mycompany
    transform:
      trace_statements:
      - context: resource
        statements:
        - replace_pattern(attributes["service.name"], "$", "/mycompany")
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
          max_recv_msg_size_mib: 20
        http:
          endpoint: ${env:MY_POD_IP}:55681
  service:
    pipelines:
      traces/tempo:
        exporters:
        - otlp/tempo
        processors:
        - memory_limiter
        - batch
        - resource
        - transform
        receivers:
        - otlp
    telemetry:
      logs:
        level: INFO
      metrics:
        address: ${env:MY_POD_IP}:8888
        level: normal
image:
  repository: harbor.mycompany.corp/dockerhub/otel/opentelemetry-collector-contrib
mode: daemonset
podLabels:
  sidecar.istio.io/inject: "true"
podMonitor:
  enabled: true
ports:
  metrics:
    enabled: true
  otlp-http:
    containerPort: 55681
    enabled: true
    hostPort: 55681
    protocol: TCP
    servicePort: 55681
resources:
  limits:
    memory: 2Gi
  requests:
    cpu: 1
    memory: 1Gi
service:
  enabled: true
serviceMonitor:
  enabled: true

kubectl get configmap -n istio-system -o jsonpath={.data.mesh} istio

accessLogEncoding: JSON
accessLogFile: /dev/stdout
accessLogFormat: |
  {
    "traceId": "%REQ(x-b3-traceid)%",
    "spanId": "%REQ(x-b3-spanid)%",
    "parentSpanId": "%REQ(x-b3-parentspanid)%"
  }
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  holdApplicationUntilProxyStarts: true
  tracing: {}
defaultProviders:
  tracing:
  - tempo
enablePrometheusMerge: true
enableTracing: true
extensionProviders:
- envoyExtAuthzHttp:
    headersToDownstreamOnDeny:
    - content-type
    - set-cookie
    headersToUpstreamOnAllow:
    - authorization
    - cookie
    - path
    - x-auth-request-user
    - x-auth-request-email
    - x-auth-request-access-token
    - x-forwarded-access-token
    includeRequestHeadersInCheck:
    - authorization
    - cookie
    - x-forwarded-access-token
    port: "80"
    service: admin-mycompany-oauth2-proxy.admin-mycompany.svc.cluster.local
    timeout: 1.5s
  name: admin-mycompany-oauth2-proxy
- envoyExtAuthzHttp:
    headersToDownstreamOnDeny:
    - content-type
    - set-cookie
    headersToUpstreamOnAllow:
    - authorization
    - cookie
    - path
    - x-auth-request-user
    - x-auth-request-email
    - x-auth-request-access-token
    - x-forwarded-access-token
    includeRequestHeadersInCheck:
    - authorization
    - cookie
    - x-forwarded-access-token
    port: "80"
    service: extra-value-admin-oauth2-proxy.extra-value-admin.svc.cluster.local
    timeout: 1.5s
  name: extra-value-admin-oauth2-proxy
- name: tempo
  opentelemetry:
    port: 4317
    service: opentelemetry-collector.opentelemetry-collector.svc.cluster.local
rootNamespace: istio-system
trustDomain: cluster.local

kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath={.data.config}
https://privatebin.net/?906fc1d81c9e5087#Btg42V4zQrskjKUtXQwPvYffGvaEeqwqEYcJEwKVHnqY

stern -n opentelemetry-collector -c opentelemetry-collector .

opentelemetry-collector-agent-z8c2w opentelemetry-collector 2024-05-08T06:26:45.129Z	info	prometheusreceiver@v0.99.0/metrics_receiver.go:340	Starting scrape manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
opentelemetry-collector-agent-z8c2w opentelemetry-collector 2024-05-08T06:26:45.129Z	info	healthcheck/handler.go:132	Health Check state change	{"kind": "extension", "name": "health_check", "status": "ready"}
opentelemetry-collector-agent-z8c2w opentelemetry-collector 2024-05-08T06:26:45.129Z	info	service@v0.99.0/service.go:192	Everything is ready. Begin running and processing data.
opentelemetry-collector-agent-z8c2w opentelemetry-collector 2024-05-08T06:26:45.129Z	warn	localhostgate/featuregate.go:63	The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}
opentelemetry-collector-agent-68cs2 opentelemetry-collector 2024-05-08T06:26:46.287Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 1}
opentelemetry-collector-agent-tghjn opentelemetry-collector 2024-05-08T06:26:47.156Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 4}
opentelemetry-collector-agent-tghjn opentelemetry-collector 2024-05-08T06:26:48.158Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 1}
opentelemetry-collector-agent-68cs2 opentelemetry-collector 2024-05-08T06:26:48.894Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 30, "data points": 32}

k get svc -n opentelemetry-collector opentelemetry-collector -o yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: opentelemetry-collector
    meta.helm.sh/release-namespace: opentelemetry-collector
  creationTimestamp: "2024-03-14T17:39:59Z"
  labels:
    app.kubernetes.io/instance: opentelemetry-collector
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opentelemetry-collector
    app.kubernetes.io/version: 0.99.0
    component: agent-collector
    helm.sh/chart: opentelemetry-collector-0.90.1
    helm.toolkit.fluxcd.io/name: opentelemetry-collector
    helm.toolkit.fluxcd.io/namespace: opentelemetry-collector
  name: opentelemetry-collector
  namespace: opentelemetry-collector
  resourceVersion: "41321246"
  uid: 6567a772-96e5-4b39-b154-77c37413f392
spec:
  clusterIP: 10.241.222.164
  clusterIPs:
  - 10.241.222.164
  internalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: jaeger-compact
    port: 6831
    protocol: UDP
    targetPort: 6831
  - name: jaeger-grpc
    port: 14250
    protocol: TCP
    targetPort: 14250
  - name: jaeger-thrift
    port: 14268
    protocol: TCP
    targetPort: 14268
  - name: metrics
    port: 8888
    protocol: TCP
    targetPort: 8888
  - appProtocol: grpc
    name: otlp
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: otlp-http
    port: 55681
    protocol: TCP
    targetPort: 55681
  - name: zipkin
    port: 9411
    protocol: TCP
    targetPort: 9411
  selector:
    app.kubernetes.io/instance: opentelemetry-collector
    app.kubernetes.io/name: opentelemetry-collector
    component: agent-collector
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Istio-gateway not generate spanId, traceId, parentSpanId #50911

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments

Select a reply

Istio-gateway not generate spanId, traceId, parentSpanId #50911

patsevanton May 8, 2024

Replies: 0 comments

patsevanton
May 8, 2024