Mixing v1 and v2 registries.conf is not rejected, surprising behavior

[fdegir]

/kind question

Description

We are working on bringing local registry for our community to avoid hitting retention and download rate limits that have recently been announced by Docker Hub. We brought up registry using Harbor and set up proxy for Docker Hub. Due to limitations of specifying proxy repo in a good way with docker client, we thought of switching to podman which is a great opportunity for us to replace docker with podman as well.

After configuring proxy on our registry, we then configured /etc/containers/registries.conf as shown below.

$ sudo cat /etc/containers/registries.conf
[registries.search]
registries = []

[registries.insecure]
registries = []

[registries.block]
registries = []

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false

When we attempt pulling an image using fully qualified name from Docker Hub through our proxy registry, things work great.
Podman correctly rewrites reference, the image gets pulled into our proxy registry and from there to the machine we're pulling the image on.

$ podman pull docker.io/hello-world
...snip...
DEBU[0000] parsed reference into "[overlay@/home/centos/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/hello-world:latest"
Trying to pull docker.io/hello-world...
DEBU[0000] reference rewritten from 'docker.io/library/hello-world:latest' to 'myregistry.com/docker-hub-proxy/library/hello-world:latest'
DEBU[0000] Trying to pull "myregistry.com/docker-hub-proxy/library/hello-world:latest"
...snip...
bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b

But when we attempt to pull the container image using unqualified name as shown below, it doesn't work.

$ podman pull hello-world
...snip...
Error: error pulling image "hello-world": unable to pull hello-world: image name provided is a short name and no search registries are defined in the registries config file.

The error message clear enough but I couldn't figure out how to get this working.

The reason for this is to avoid changing existing Dockerfiles that use unqualified names and instead do stuff with podman on build servers, transparent from users.

Steps to reproduce the issue:

1. Update /etc/containers/registries.conf similar to below

[registries.search]
registries = []

[registries.insecure]
registries = []

[registries.block]
registries = []

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false

2. Attempt to pull an image using qualified name which should work

$ podman pull docker.io/hello-world

3. Attempt to pull the same image using unqualified name which doesn't work

$ podman pull hello-world

Describe the results you received:
Pulling image with unqualified name doesn't work.

Describe the results you expected:
Pulling image with unqualified name works.

Additional information you deem important (e.g. issue happens only occasionally):
This may be an error on our part but we couldn't figure out how to achieve this by following documentation or if this is at all possible.

Output of podman version:

$ podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.13.4
  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.2.0+305+5e198a41.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: a2b11288060ebd7abd20e0b4eb1a834bbf0aec3e'
  Distribution:
    distribution: '"centos"'
    version: "8"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 4256825344
  MemTotal: 8191909888
  OCIRuntime:
    name: runc
    package: runc-1.0.0-65.rc10.module_el8.2.0+305+5e198a41.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: eiffel.build.novalocal
  kernel: 4.18.0-147.3.1.el8_1.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.2-3.git21fdece.module_el8.2.0+305+5e198a41.x86_64
    Version: |-
      slirp4netns version 0.4.2+dev
      commit: 21fdece2737dc24ffa3f01a341b8a6854f8b13b4
  uptime: 4h 8m 38.49s (Approximately 0.17 days)
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /home/centos/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.2-5.module_el8.2.0+305+5e198a41.x86_64
      Version: |-
        fuse-overlayfs: version 0.7.2
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  GraphRoot: /home/centos/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 10
  RunRoot: /run/user/1000
  VolumePath: /home/centos/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-1.6.4-10.module_el8.2.0+305+5e198a41.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

On Ubuntu18.04 with version 2.1.0 and the result is the same.

Additional environment details (AWS, VirtualBox, physical, etc.):
The nodes where we attempted this are provisioned from OpenStack.
The attempts were made on Centos8 and Ubuntu18.04 instances.

[openshift-ci-robot]

@fdegir: The label(s) kind/question cannot be applied, because the repository doesn't have them

In response to this:

/kind question

Description

We are working on bringing local registry for our community to avoid hitting retention and download rate limits that have recently been announced by Docker Hub. We brought up registry using Harbor and set up proxy for Docker Hub. Due to limitations of specifying proxy repo in a good way with docker client, we thought of switching to podman which is a great opportunity for us to replace docker with podman as well.

After configuring proxy on our registry, we then configured /etc/containers/registries.conf as shown below.

$ sudo cat /etc/containers/registries.conf
[registries.search]
registries = []

[registries.insecure]
registries = []

[registries.block]
registries = []

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false

When we attempt pulling an image using fully qualified name from Docker Hub through our proxy registry, things work great.
Podman correctly rewrites reference, the image gets pulled into our proxy registry and from there to the machine we're pulling the image on.

$ podman pull docker.io/hello-world
...snip...
DEBU[0000] parsed reference into "[overlay@/home/centos/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/hello-world:latest"
Trying to pull docker.io/hello-world...
DEBU[0000] reference rewritten from 'docker.io/library/hello-world:latest' to 'myregistry.com/docker-hub-proxy/library/hello-world:latest'
DEBU[0000] Trying to pull "myregistry.com/docker-hub-proxy/library/hello-world:latest"
...snip...
bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b

But when we attempt to pull the container image using unqualified name as shown below, it doesn't work.

$ podman pull docker.io/hello-world
...snip...
Error: error pulling image "hello-world": unable to pull hello-world: image name provided is a short name and no search registries are defined in the registries config file.

The error message clear enough but I couldn't figure out how to get this working.

The reason for this is to avoid changing existing Dockerfiles that use unqualified names and instead do stuff with podman on build servers, transparent from users.

Steps to reproduce the issue:

1. Update /etc/containers/registries.conf similar to below

[registries.search]
registries = []

[registries.insecure]
registries = []

[registries.block]
registries = []

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false

2. Attempt to pull an image using qualified name which should work

$ podman pull docker.io/hello-world

3. Attempt to pull the same image using unqualified name which doesn't work

$ podman pull hello-world

Describe the results you received:
Pulling image with unqualified name doesn't work.

Describe the results you expected:
Pulling image with unqualified name works.

Additional information you deem important (e.g. issue happens only occasionally):
This may be an error our part but we couldn't figure out how to achieve this by following documentation or if this is at all possible.

Output of podman version:

$ podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.13.4
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
 compiler: gc
 git commit: ""
 go version: go1.13.4
 podman version: 1.6.4
host:
 BuildahVersion: 1.12.0-dev
 CgroupVersion: v1
 Conmon:
   package: conmon-2.0.6-1.module_el8.2.0+305+5e198a41.x86_64
   path: /usr/bin/conmon
   version: 'conmon version 2.0.6, commit: a2b11288060ebd7abd20e0b4eb1a834bbf0aec3e'
 Distribution:
   distribution: '"centos"'
   version: "8"
 IDMappings:
   gidmap:
   - container_id: 0
     host_id: 1000
     size: 1
   - container_id: 1
     host_id: 100000
     size: 65536
   uidmap:
   - container_id: 0
     host_id: 1000
     size: 1
   - container_id: 1
     host_id: 100000
     size: 65536
 MemFree: 4256825344
 MemTotal: 8191909888
 OCIRuntime:
   name: runc
   package: runc-1.0.0-65.rc10.module_el8.2.0+305+5e198a41.x86_64
   path: /usr/bin/runc
   version: 'runc version spec: 1.0.1-dev'
 SwapFree: 0
 SwapTotal: 0
 arch: amd64
 cpus: 4
 eventlogger: journald
 hostname: eiffel.build.novalocal
 kernel: 4.18.0-147.3.1.el8_1.x86_64
 os: linux
 rootless: true
 slirp4netns:
   Executable: /usr/bin/slirp4netns
   Package: slirp4netns-0.4.2-3.git21fdece.module_el8.2.0+305+5e198a41.x86_64
   Version: |-
     slirp4netns version 0.4.2+dev
     commit: 21fdece2737dc24ffa3f01a341b8a6854f8b13b4
 uptime: 4h 8m 38.49s (Approximately 0.17 days)
registries:
 blocked: null
 insecure: null
 search: null
store:
 ConfigFile: /home/centos/.config/containers/storage.conf
 ContainerStore:
   number: 0
 GraphDriverName: overlay
 GraphOptions:
   overlay.mount_program:
     Executable: /usr/bin/fuse-overlayfs
     Package: fuse-overlayfs-0.7.2-5.module_el8.2.0+305+5e198a41.x86_64
     Version: |-
       fuse-overlayfs: version 0.7.2
       FUSE library version 3.2.1
       using FUSE kernel interface version 7.26
 GraphRoot: /home/centos/.local/share/containers/storage
 GraphStatus:
   Backing Filesystem: xfs
   Native Overlay Diff: "false"
   Supports d_type: "true"
   Using metacopy: "false"
 ImageStore:
   number: 10
 RunRoot: /run/user/1000
 VolumePath: /home/centos/.local/share/containers/storage/volumes```

**Package info (e.g. output of `rpm -q podman` or `apt list podman`):**

$ rpm -q podman
podman-1.6.4-10.module_el8.2.0+305+5e198a41.x86_64

**Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?**

Yes

On Ubuntu18.04 with version 2.1.0 and the result is the same.

**Additional environment details (AWS, VirtualBox, physical, etc.):**
The nodes where we attempted this are provisioned from OpenStack.
The attempts were made on Centos8 and Ubuntu18.04 instances.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rhatdan]

@vrothberg @mtrmac @QiWang19 PTAL

[mtrmac]

@fdegir The configuration file is mixing up v1 and v2 syntax; drop all lines before the unqualified-search-registries line.

That will make it work, but the way it fails is surprising to me — I can’t see how we can end up using the v1 search value while using the v2 registries list; AFAICS the code should refuse to work on mixed v1/v2 files completely.

[fdegir]

Thanks a lot for the hint @mtrmac. I can confirm that dropping lines before unqualified-search-registries makes it work and we can pull the image using unqualified name from our proxy.

Regarding mixing v1/v2; podman fails with the correct error message if v1 registries.search contains element(s). but if registries.search is empty, the error message is different which is perhaps what confused us.

I include these 3 different cases below for the sake of completion. Please note that these are tested on Centos8 with podman version 1.6.4.

v1/v2 mixed with registries.search containining an element.
podman pull of an image using unqualified name fails as expected with the correct error message.

$ cat /etc/containers/registries.conf
[registries.search]
registries = ["docker.io"]

[registries.insecure]
registries = []

[registries.block]
registries = []

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false
$ podman pull hello-world
Error: error pulling image "hello-world": unable to pull hello-world: error getting default registries to try: mixing sysregistry v1/v2 is not supported

v1/v2 mixed with an empty registries.search.
podman pull of an image using unqualified name fails but with a misleading error message.

$ cat /etc/containers/registries.conf
[registries.search]
registries = []

[registries.insecure]
registries = []

[registries.block]
registries = []

unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false
$ podman pull hello-world
Error: error pulling image "hello-world": unable to pull hello-world: image name provided is a short name and no search registries are defined in the registries config file.

v2 only.
podman pull of an image using unqualified name works.

$ cat /etc/containers/registries.conf
unqualified-search-registries = ["docker.io"]

[[registry]]
prefix = "docker.io"
location = "myregistry.com/docker-hub-proxy"
insecure = false
$ podman pull hello-world
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 skipped: already exists
Copying config bf756fb1ae done
Writing manifest to image destination
Storing signatures
bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b

I suppose this issue can either be closed or kept open/turned into bug report if you think it is appropriate to do so.

Thanks for the help.

[mtrmac]

Reopening to see what the parser is doing — and moving to c/image.

[mtrmac]

v1/v2 mixed with an empty registries.search.
podman pull of an image using unqualified name fails but with a misleading error message.

The current version warns, in debug mode only:

DEBU[0000] Failed to decode keys ["registries.block.unqualified-search-registries"] from "…"

Per #1507 , we don’t dare to make this a more visible error. So, that part is not going to get significantly better.

But the recognized empty registries = [] blocks are a clear enough indication of a confused file, and we can detect that part: #1753 .