bake: default environment variable takes over override

[asvinours]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When defining a variable with a default value coming from the environment, bake will always take the value from the environment variable even if another bake file defines an override. The only way for the override to be used is for the environment variable to be entirely absent (setting it to an empty value is not enough)

Expected behaviour

The override should take precedence over the default value

Actual behaviour

the value from the environment variable is used

Buildx version

github.com/docker/buildx v0.14.0 171fcbe

Docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1018-aws
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: ip-10-13-2-22
 ID: 2180b260-4845-4a4a-9801-cf350854cac6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker
 \_ default    \_ default       running   v0.13.1    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Configuration

# docker-bake-base.hcl

variable "IMAGE_NAME" {
  default="$IMAGE_NAME"
}

variable "AWS_ACCOUNT_ID" {
  default = "$AWS_ACCOUNT_ID"
}

variable "AWS_REGION" {
  default = "$AWS_REGION"
}

target "default" {
  output = [
    "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${IMAGE_NAME}",
  ]
}

---

# docker-bake.hcl

IMAGE_NAME="john"

target "my-target" {
  inherits = ["default"]

}

---

IMAGE_NAME="bob" docker buildx bake -f docker-bake-base.hcl -f 'docker-bake.hcl' --print my-target
{
  "group": {
    "default": {
      "targets": [
        "my-target"
      ]
    }
  },
  "target": {
    "my-target": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "output": [
        "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/bob"
      ]
    }
  }
}

IMAGE_NAME="" docker buildx bake -f docker-bake-base.hcl -f 'docker-bake.hcl' --print my-target
{
  "group": {
    "default": {
      "targets": [
        "my-target"
      ]
    }
  },
  "target": {
    "my-target": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "output": [
        "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/"
      ]
    }
  }
}

docker buildx bake -f docker-bake-base.hcl -f 'docker-bake.hcl' --print my-target
{
  "group": {
    "default": {
      "targets": [
        "my-target"
      ]
    }
  },
  "target": {
    "my-target": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "output": [
        "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/john"
      ]
    }
  }
}

Build logs

No response

Additional info

No response

[tonistiigi]

That looks like expected behavior with the correct order of precedence. Configuration passed in from env and flags overrides the config loaded from files. If you worry about collisions, you should define a more unique name, eg. BUILDX_MYAPP_IMAGE_NAME. If you don't want your arg to be set from environment then don't define "variable" block and only define the attribute in the hcl definition.

[asvinours]

I see. I think I see where the confusion came from for me: reading the use environment variable as default documentation, I had understood that for an environment variable to be used, the bake variable default value had to be set to the name of an environment variable.

Doing more testing, I realised that I was wrong.
Setting the default value of a bake variable to an environment variable name doesn't seem to have much effect in itself.
What does, seem to be that the environment variable name has to match the bake variable name and the attribute is defined with a variable block:

variable "IMAGE_NAME" {
  default="$MY_IMAGE_NAME"
}

target "default" {
  output = [
    "123456123456.dkr.ecr.us-east-1.amazonaws.com/${IMAGE_NAME}",
  ]
}

export MY_IMAGE_NAME="bob"; 
docker buildx bake -f docker-bake.hcl --print
{
  "target": {
    "default": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "output": [
        "123456123456.dkr.ecr.us-east-1.amazonaws.com/$MY_IMAGE_NAME"
      ]
    }
  }
}

On the other hand:

variable "IMAGE_NAME" {
  default="hardcoded-default-name"
}

target "default" {
  output = [
    "123456123456.dkr.ecr.us-east-1.amazonaws.com/${IMAGE_NAME}",
  ]
}

export IMAGE_NAME="bob"; 
docker buildx bake -f docker-bake-base.hcl --print
{
  "target": {
    "default": {
      "context": ".",
      "dockerfile": "Dockerfile",
      "output": [
        "123456123456.dkr.ecr.us-east-1.amazonaws.com/bob"
      ]
    }
  }
}

As you mentioned, to prevent bake from reading from the environment variables, I can define the attribute without the variable block.