"OTEL_RESOURCE_ATTRIBUTES" in .env is ignored when using docker compose

[mrasu]

Description

When using docker compose, value of "OTEL_RESOURCE_ATTRIBUTES" specifyied at .env file is ignored and the value becomes docker.cli.cobra.command_path=docker%20compose instead.
It seems #4875 changes behahavior to override OTEL_RESOURCE_ATTRIBUTES.

use-case

opentelemetry-demo uses .env to specify service.namespace for otel's namespace.
But it is not working now.

You can see code here

Reproduce

1. Create .env file specifing OTEL_RESOURCE_ATTRIBUTES

OTEL_RESOURCE_ATTRIBUTES="service.namespace=opentelemetry-demo"

2. Create docker-compose.yml using OTEL_RESOURCE_ATTRIBUTES

services:
  something:
    image: ubuntu
    command: tail -f /dev/null
    environment:
      - OTEL_RESOURCE_ATTRIBUTES

3. docker compose up
4. Go into "something"
5. Check ${OTEL_RESOURCE_ATTRIBUTES}
6. You will see the value is docker.cli.cobra.command_path=docker%20compose instead of service.namespace=opentelemetry-demo

Expected behavior

Keep the value of "OTEL_RESOURCE_ATTRIBUTES" in .env.
I believe adding docker.cli.cobra.command_path attribute is fine but the original value should be kept

docker version

Client: Docker Engine - Community
 Cloud integration: v1.0.24
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.5.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.17.0
    Path:     /usr/lib/docker/cli-plugins/docker-scan

Server:
 Containers: 199
  Running: 1
  Paused: 0
  Stopped: 198
 Images: 2751
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-26-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 46.34GiB
 Name: hiroaki
 ID: UHYB:2Z3Y:2UOI:65QQ:5RSA:GUY5:IIIY:CLDG:M5ED:EW2D:43T2:6VNF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[mrasu]

any chance?

[laurazard]

Hiya @mrasu, sorry for the delay in replying –

Indeed, we started setting OTEL_RESOURCE_ATTRIBUTES which would break setups such as yours. This wasn't the intention, and I guess there's a few ways we could take to avoid collisions here: we could use a separate/custom env var for our instrumenting since we can make sure both ends look at the same place, or we could check for the env var on the CLI and append our attribute instead of just overwriting.
I'm open to opinions here (cc @milas @jsternberg @krissetto).

In the meantime, for your purposes, you could set your .env such as

MY_CUSTOM_ENV_VAR="service.namespace=opentelemetry-demo"

and your compose file such as

services:
  something:
    image: ubuntu
    command: tail -f /dev/null
    environment:
      OTEL_RESOURCE_ATTRIBUTES: ${MY_CUSTOM_ENV_VAR}

[thaJeztah]

Hm.. right, so;

1. .env (again)
2. compose as CLI plugin, so may inherit env-vars that are set by docker CLI (and docker CLI doesn't use .env, so wouldn't even be aware the user is about to set SOME_ENV_VAR)
3. we don't check if OTEL_RESOURCE_ATTRIBUTES is set already (in either case), and assume it's not set, therefore overwriting any user-set option for OTEL?

Wondering if this one should either detect if it’s already set, or prepend the env-var instead of appending it;

cli/cli-plugins/manager/cobra.go

Lines 136 to 148 in b6c5522

	func appendPluginResourceAttributesEnvvar(env []string, cmd *cobra.Command, plugin Plugin) []string {
	if attrs := getPluginResourceAttributes(cmd, plugin); attrs.Len() > 0 {
	// values in environment variables need to be in baggage format
	// otel/baggage package can be used after update to v1.22, currently it encodes incorrectly
	attrsSlice := make([]string, attrs.Len())
	for iter := attrs.Iter(); iter.Next(); {
	i, v := iter.IndexedAttribute()
	attrsSlice[i] = string(v.Key) + "=" + url.PathEscape(v.Value.AsString())
	}
	env = append(env, ResourceAttributesEnvvar+"="+strings.Join(attrsSlice, ","))
	}
	return env
	}

I guess the overall issue to look for is;

• decide how we can best handle OTEL configuration (standard for OTEL is to use env-vars, but for the CLI that's sometimes more tricky, and I know that for some setups, we want to be able to set-up through configuration file(s), which can be depending on the docker context in use.
• but OTEL does allow configuration through env-vars, which means that in an environment where OTEL is already set-up we must be able to "play nice"
• ☝️ which could either mean; don't do anything if we detect anything OTEL-related in env-vars (but the list of OTEL-related env-vars may evolve over time)
• ☝️ or ... merge configs (but this may result in some franken-config 🧟 of defaults we set and those that are set in the environment