lazy pull from private registry #1107

reisman234 · 2023-02-13T21:41:25Z

Hi,
I have now a setup with rootless podman and the stargz-store running, which works with the pre-converted-images
The stargz-store creates for that pull the following log messages.

{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:41c27163b036ab20bb586468eb392dcccf09b601936876ceeb9ef9eb771410b7","time":"2023-02-13T21:04:35.622823154+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:a9de9abd7421b69365510423dbdb98e3a9f9b1c88553847cd1cd17bea694a9d5","time":"2023-02-13T21:04:35.626255541+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:34575e03199f152f93f090f9d772e4daeb698ff7edce98c5f75c386b0bce2c84","time":"2023-02-13T21:04:35.626366655+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:e64f47341297137687153c7a28d0c2c265c1bb85f1ec5bacbb730033868de913","time":"2023-02-13T21:04:35.629803426+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:05718b65241b666f931638eb279c8d4f45f47f092e483386138fab55ba3ffd2b","time":"2023-02-13T21:04:35.630207610+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:55c6cce41d1d0053dd4f2d9a1cd89587e51d82b67ddd02548e61e4767792a565","time":"2023-02-13T21:04:35.630680451+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:720d3b898d042dc38f76d7a27550dd4165bdebf127fc5c0edf86209d49e6224b","time":"2023-02-13T21:04:35.637081607+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:654ab7983ba6e38c45cef81a3f28c5c58cae21160d13c59fa40187873682dcf8","time":"2023-02-13T21:04:35.637981816+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"ghcr.io/stargz-containers/python:3.10-esgz/sha256:64f9e3b44d1e96f33260519a189bf246a2294d4c857bb21716543e7ca540678b","time":"2023-02-13T21:04:35.643907679+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:654ab7983ba6e38c45cef81a3f28c5c58cae21160d13c59fa40187873682dcf8 is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.494205924+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:720d3b898d042dc38f76d7a27550dd4165bdebf127fc5c0edf86209d49e6224b is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.901575841+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:e64f47341297137687153c7a28d0c2c265c1bb85f1ec5bacbb730033868de913 is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.902063791+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:55c6cce41d1d0053dd4f2d9a1cd89587e51d82b67ddd02548e61e4767792a565 is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.902486843+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:41c27163b036ab20bb586468eb392dcccf09b601936876ceeb9ef9eb771410b7 is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.902778648+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:05718b65241b666f931638eb279c8d4f45f47f092e483386138fab55ba3ffd2b is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.903160820+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:64f9e3b44d1e96f33260519a189bf246a2294d4c857bb21716543e7ca540678b is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.903445017+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:34575e03199f152f93f090f9d772e4daeb698ff7edce98c5f75c386b0bce2c84 is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.903704517+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:a9de9abd7421b69365510423dbdb98e3a9f9b1c88553847cd1cd17bea694a9d5 is marked as USING","refcounter":1,"time":"2023-02-13T21:04:36.903972627+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:41c27163b036ab20bb586468eb392dcccf09b601936876ceeb9ef9eb771410b7 is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.904363132+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:720d3b898d042dc38f76d7a27550dd4165bdebf127fc5c0edf86209d49e6224b is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.916312308+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:55c6cce41d1d0053dd4f2d9a1cd89587e51d82b67ddd02548e61e4767792a565 is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.918885566+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:e64f47341297137687153c7a28d0c2c265c1bb85f1ec5bacbb730033868de913 is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.921507330+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:05718b65241b666f931638eb279c8d4f45f47f092e483386138fab55ba3ffd2b is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.924042146+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:654ab7983ba6e38c45cef81a3f28c5c58cae21160d13c59fa40187873682dcf8 is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.926691244+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:64f9e3b44d1e96f33260519a189bf246a2294d4c857bb21716543e7ca540678b is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.929220738+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:34575e03199f152f93f090f9d772e4daeb698ff7edce98c5f75c386b0bce2c84 is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.931653753+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz / sha256:a9de9abd7421b69365510423dbdb98e3a9f9b1c88553847cd1cd17bea694a9d5 is marked as USING","refcounter":2,"time":"2023-02-13T21:04:36.934205805+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:654ab7983ba6e38c45cef81a3f28c5c58cae21160d13c59fa40187873682dcf8 is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.952858137+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:720d3b898d042dc38f76d7a27550dd4165bdebf127fc5c0edf86209d49e6224b is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.952883188+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:e64f47341297137687153c7a28d0c2c265c1bb85f1ec5bacbb730033868de913 is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.952947176+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:55c6cce41d1d0053dd4f2d9a1cd89587e51d82b67ddd02548e61e4767792a565 is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.953054586+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:64f9e3b44d1e96f33260519a189bf246a2294d4c857bb21716543e7ca540678b is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.953085576+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:34575e03199f152f93f090f9d772e4daeb698ff7edce98c5f75c386b0bce2c84 is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.953111009+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:41c27163b036ab20bb586468eb392dcccf09b601936876ceeb9ef9eb771410b7 is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.953169100+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:05718b65241b666f931638eb279c8d4f45f47f092e483386138fab55ba3ffd2b is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.953187698+01:00"}
{"level":"info","msg":"layer ghcr.io/stargz-containers/python:3.10-esgz/sha256:a9de9abd7421b69365510423dbdb98e3a9f9b1c88553847cd1cd17bea694a9d5 is marked as RELEASE","refcounter":1,"time":"2023-02-13T21:04:36.953200541+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969709095+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969746344+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969825982+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969851330+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969869493+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969930477+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969952357+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.969966069+01:00"}
{"level":"warning","msg":"unknown filename \"diff1\"","time":"2023-02-13T21:04:36.970027274+01:00"}

But when I pull my own created Image this will not work properly

{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:c88bd64aa02fa0566af3d09d87effc9abe5b9527a8efddf8b3d3c5723f1767e0","time":"2023-02-13T22:22:26.267748649+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:29f91329f99f24e95de00e8d06f28b32ad2c7ff28ffac134ac0fc4df77789806","time":"2023-02-13T22:22:26.279738951+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:467c5dda7d0e2ba52e0672b7785b9ddef5a4d5fa9277732e6e9149314f6ef0d8","time":"2023-02-13T22:22:26.281220107+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:18863545c16e6c1099d4d0ff7fe736f74562e6630e83bfcbdf0bdd0f94775534","time":"2023-02-13T22:22:26.437406550+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:bdb549823857a1214b64ae2c650cf76a0d410a2b04b6367cbb4528f27d0b4a79","time":"2023-02-13T22:22:26.438712893+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:28e2ad918b725ffe8c3433ff68ce483dac7a0f198b91f03cf28e0d1a0436455b","time":"2023-02-13T22:22:26.441863437+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:0fca9493446747c40bb5a5601255f18d503f5f36d685958992a3ea4320c0c73d","time":"2023-02-13T22:22:26.442773305+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:13355b4dff2553b74016fdf7eb66637db2e24ece63bfb7072e4c024cde8fd947","time":"2023-02-13T22:22:26.444860430+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:7a8c2176b630fdba501fc8778b3f0edebc46b38669ac18836b1bb681441f9cbd","time":"2023-02-13T22:22:26.446628013+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:758dd67c0aae6564403dcd6e6abe51e26f78afbdebc10d7603e5f3d79c3ca966","time":"2023-02-13T22:22:26.448429380+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz/sha256:a8f2b60b399869d8ce4c63d8765fb6f6ef06e1ec8957ac1c43a957cf291aba4c","time":"2023-02-13T22:22:26.449820730+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:29f91329f99f24e95de00e8d06f28b32ad2c7ff28ffac134ac0fc4df77789806\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:29f91329f99f24e95de00e8d06f28b32ad2c7ff28ffac134ac0fc4df77789806\"","time":"2023-02-13T22:22:27.012134234+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:758dd67c0aae6564403dcd6e6abe51e26f78afbdebc10d7603e5f3d79c3ca966\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:758dd67c0aae6564403dcd6e6abe51e26f78afbdebc10d7603e5f3d79c3ca966\"","time":"2023-02-13T22:22:27.047778105+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:28e2ad918b725ffe8c3433ff68ce483dac7a0f198b91f03cf28e0d1a0436455b\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:28e2ad918b725ffe8c3433ff68ce483dac7a0f198b91f03cf28e0d1a0436455b\"","time":"2023-02-13T22:22:27.099642832+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:c88bd64aa02fa0566af3d09d87effc9abe5b9527a8efddf8b3d3c5723f1767e0\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:c88bd64aa02fa0566af3d09d87effc9abe5b9527a8efddf8b3d3c5723f1767e0\"","time":"2023-02-13T22:22:27.102848658+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:7a8c2176b630fdba501fc8778b3f0edebc46b38669ac18836b1bb681441f9cbd\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:7a8c2176b630fdba501fc8778b3f0edebc46b38669ac18836b1bb681441f9cbd\"","time":"2023-02-13T22:22:27.104807692+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:a8f2b60b399869d8ce4c63d8765fb6f6ef06e1ec8957ac1c43a957cf291aba4c\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:a8f2b60b399869d8ce4c63d8765fb6f6ef06e1ec8957ac1c43a957cf291aba4c\"","time":"2023-02-13T22:22:27.109922544+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:467c5dda7d0e2ba52e0672b7785b9ddef5a4d5fa9277732e6e9149314f6ef0d8\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:467c5dda7d0e2ba52e0672b7785b9ddef5a4d5fa9277732e6e9149314f6ef0d8\"","time":"2023-02-13T22:22:27.110627628+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:0fca9493446747c40bb5a5601255f18d503f5f36d685958992a3ea4320c0c73d\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:0fca9493446747c40bb5a5601255f18d503f5f36d685958992a3ea4320c0c73d\"","time":"2023-02-13T22:22:27.111292098+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:18863545c16e6c1099d4d0ff7fe736f74562e6630e83bfcbdf0bdd0f94775534\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:18863545c16e6c1099d4d0ff7fe736f74562e6630e83bfcbdf0bdd0f94775534\"","time":"2023-02-13T22:22:27.120081497+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:bdb549823857a1214b64ae2c650cf76a0d410a2b04b6367cbb4528f27d0b4a79\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:bdb549823857a1214b64ae2c650cf76a0d410a2b04b6367cbb4528f27d0b4a79\"","time":"2023-02-13T22:22:27.120657513+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/cuda:11.7.0-devel-ubuntu20.04-esgz\" / \"sha256:13355b4dff2553b74016fdf7eb66637db2e24ece63bfb7072e4c024cde8fd947\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:13355b4dff2553b74016fdf7eb66637db2e24ece63bfb7072e4c024cde8fd947\"","time":"2023-02-13T22:22:27.179867344+01:00"}

After that, I pushed one of the pre-converted-images in my private registry and tried to pull that, but it end with similar messages.

{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:21.836546343+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:21.850375755+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:21.853963611+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:21.965204203+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:21.966535882+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:21.968868275+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:21.970761984+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:21.972706522+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:21.977061323+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac\"","time":"2023-02-13T22:29:22.225680706+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:22.397320009+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:22.400293523+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:22.401729187+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:22.405262611+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:22.406599119+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:22.409238035+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:22.411215055+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:22.416634178+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:22.418582546+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645\"","time":"2023-02-13T22:29:22.445800030+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:22.753129195+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:22.753204045+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:22.754413201+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:22.756588947+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:22.760163310+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:22.763277316+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:23.007283894+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:23.063887388+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:23.063942179+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8\"","time":"2023-02-13T22:29:23.131484976+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:23.156664600+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:23.272762268+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:23.274885278+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:23.278072460+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:23.281073872+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:23.292488638+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:23.293127749+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:23.293821710+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:23.294461982+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae\"","time":"2023-02-13T22:29:23.571547444+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:23.599068006+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:23.607934735+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:23.613319150+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:23.616714669+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:23.744769182+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:23.746593620+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:23.752186919+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:23.754823908+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:23.763456805+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25\"","time":"2023-02-13T22:29:23.833110552+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:24.042550327+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:24.044998955+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:24.066624753+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:24.068094794+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:24.071803533+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:24.073832409+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:24.076207095+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:24.079464852+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:24.320180076+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8\"","time":"2023-02-13T22:29:24.378124848+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:24.568292185+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:24.577115157+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:24.579260459+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:24.581639670+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:24.623256709+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:24.651776144+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:24.712764681+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220\"","time":"2023-02-13T22:29:24.795395450+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:24.860264804+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:24.945411780+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:24.961994187+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:24.967945126+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:24.967998321+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:25.005489334+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:25.075642914+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:25.079107756+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:25.083392176+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3\"","time":"2023-02-13T22:29:25.278088108+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:25.335731530+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:744c9ddedbcf06f0d5837d90e5201f1250a174ca0b51157ece74fd85897b5dae","time":"2023-02-13T22:29:25.449078977+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:90b6cbfd1ad5900542743c27996465fd645242832a8e9b0fd75d1efbc18e6220","time":"2023-02-13T22:29:25.462395924+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2bb4fdcfe2203389fbfdebf83b3308ff9ef0117fae476d4cd80fe34d93d8cce3","time":"2023-02-13T22:29:25.465765107+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:5b508727f4b3a235d58a4000e80fe7ca1e1578d0efe465e63c5ff5f00f8fe6ac","time":"2023-02-13T22:29:25.468044172+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:090307217139be503868c1b4b2cfc4952d7aa0764bbf2fd1326fe4601c1d98f8","time":"2023-02-13T22:29:25.651131762+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:d4b9c1c30162a29f619ad1d6598dfa3ae5c6898816d2c90dad98463bcaeb7d25","time":"2023-02-13T22:29:25.652533470+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:25.654337607+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2","time":"2023-02-13T22:29:25.656309567+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:4e1927ed47969127e03692563d737fc850cc7eaa6e10ae84646170f955101645","time":"2023-02-13T22:29:25.710535296+01:00"}
{"error":"failed to resolve layer: failed to resolve layer \"REGISTRY:library/python:3.10-esgz\" / \"sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"harbor.gx4ki.imla.hs-offenburg.de\", ref:\"REGISTRY:library/python:3.10-esgz\", digest:\"sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2\"): failed to access to the registry with code 401: failed to resolve","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:2dd87d4541cdaabb777d66a6f325aa5e5ae50ad7280263c35b9055eb2faca9e2\"","time":"2023-02-13T22:29:25.886565363+01:00"}
{"level":"info","msg":"Received status code: 401 Unauthorized. Refreshing creds...","src":"REGISTRY:library/python:3.10-esgz/sha256:33bb2d16c2f0f551597c72b8e61706f109027adf08acc71c2cd2d86665f21bd8","time":"2023-02-13T22:29:26.042668986+01:00"}

Because of the 401 Unauthorized Error, I did also switch the accessibility of that specific repository to public, but with the same result.

Thanks for any help on this!

The text was updated successfully, but these errors were encountered:

Dadavan · 2023-03-19T23:00:56Z

We are experiencing the same issue. We have configured stargz-snapshotter with containerd as described in the docs. The supplied pre built images are pulling lazily with no issues (can confirm from logs and also can see the fuse mount), but when pushing the pre built image to our own Docker repository it fails to pull lazily and prints the following error (this happens also when the repository is public, not just private):

Mar 19 22:48:33 ip-10-101-48-29 containerd-stargz-grpc[19605]: {"key":"k8s.io/203/extract-575453287-y9WT sha256:c00e77fc513e8777043d58387a011deb334b3976cc32445e2df11343a670d034","level":"info","mountpoint":"/var/lib/containerd/io.containerd.snapshotter.v1.stargz/snapshotter/snapshots/152/fs","msg":"Received status code: 401 Unauthorized. Refreshing creds...","parent":"","src":"docker.io/dadavan/python:3.10-esgz/sha256:918438fbf26b133ff20f857054d8ef6e7992fce12bd77d7b058ad89d739994c5","time":"2023-03-19T22:48:33.631845435Z"}
Mar 19 22:48:33 ip-10-101-48-29 containerd-stargz-grpc[19605]: {"error":"failed to resolve layer: failed to resolve layer \"sha256:5aadac84e9a45bee7a1bdc7a69c14245c9c2a858c38141a8da6270ae9e463146\" from \"docker.io/dadavan/python:3.10-esgz\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header: failed to resolve target","key":"k8s.io/203/extract-575453287-y9WT sha256:c00e77fc513e8777043d58387a011deb334b3976cc32445e2df11343a670d034","level":"warning","msg":"failed to prepare remote snapshot","parent":"","remote-snapshot-prepared":"false","time":"2023-03-19T22:48:33.902988919Z"}

We have configured CRI based authentication as described here.

containerd config:

version = 2

# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
# The specified address below is the default which stargz snapshotter listen to.
[proxy_plugins.stargz]
type = "snapshot"
address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.6@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db"
max_concurrent_downloads = 9

[plugins."io.containerd.grpc.v1.cri".containerd]
# Use stargz snapshotter through CRI
snapshotter = "stargz"
disable_snapshot_annotations = false

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

stargz config.toml:

# Stargz Snapshotter proxies CRI Image Service into containerd socket.
[cri_keychain]
enable_keychain = true
image_service_path = "/run/containerd/containerd.sock"

stargz process is running with the following flags:

/usr/bin/containerd-stargz-grpc --root=/var/lib/containerd/io.containerd.snapshotter.v1.stargz --config=/etc/containerd-stargz-grpc/config.toml

And kubelet is running with the --image-service-endpoint=unix:///run/containerd-stargz-grpc/containerd-stargz-grpc.sock flag as instructed.

stargz-snapshotter service has been restarted, followed by containerd and finally kubelet.

Is there anything wrong with our setup or configuration? We really are out of ideas here and would appreciate your help.

Thank you very much

ktock · 2023-03-20T01:04:57Z

docker.io/dadavan/python:3.10-esgz": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header: failed to resolve target

@Dadavan Looks like the image isn't formatted as eStargz. Could you provide the commands used for creating and pushing the image?

Dadavan · 2023-03-20T10:22:20Z

docker.io/dadavan/python:3.10-esgz": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header: failed to resolve target

@Dadavan Looks like the image isn't formatted as eStargz. Could you provide the commands used for creating and pushing the image?

In the above case I simply pulled one of the pre built images, tagged it and pushed to my own repo:

docker pull ghcr.io/stargz-containers/python:3.10-esgz
docker tag ghcr.io/stargz-containers/python:3.10-esgz dadavan/python:3.10-esgz
docker push dadavan/python:3.10-esgz

We did also try building our own images but encountered the same problem, in that case the command we used was:

docker buildx build -t bringg/hive:esgz -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true .

ktock · 2023-03-21T11:11:06Z

@Dadavan

docker

Does using other image tools (e.g. crane) or other registry (e.g. ghcr.io) solve the issue?

docker buildx build -t bringg/hive:esgz -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true .

This should work even with Docker Hub. Which BuildKit version are you using? BuildKit >= v0.10 is needed.

Dadavan · 2023-03-30T08:25:46Z

@Dadavan

docker

Does using other image tools (e.g. crane) or other registry (e.g. ghcr.io) solve the issue?

docker buildx build -t bringg/hive:esgz -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true .

This should work even with Docker Hub. Which BuildKit version are you using? BuildKit >= v0.10 is needed.

I tried performing the same steps with nerdctl (pull, tag, push to dockerhub) and it does work, problem is we can't use nerdctl in our automation pipeline currently. buildx version is v0.10.3.

hagaibarel · 2023-09-14T07:01:03Z

Hi folks, anything new with this?

dmosdallas · 2023-09-20T14:26:21Z

@hagaibarel, I think there were two different issues in this thread, not sure which one you're looking for updates on:

The gzip errors which looks like it's due to the built image not being in the expected eStargz format. That may have been due to using a too-old version of the builder? Kind of unclear.
The 401 errors from the registry. I had a similar issue (Images pushed to private repository not fetched lazily #1389) that was resolved by sharing my docker credentials with the snapshotter

When using the buildx build command above and configuring the snapshotter to use my user's docker credentials, I've been able to successfully build and lazily pull eStargz images with a private repo. Happy to help troubleshoot if you're still stuck here.

sairameshv · 2024-04-17T09:50:07Z

Hello !!

I'm trying out this Snapshotter (Stargz-Store) service and ended up in a similar situation.
All the prebuilt images are being pulled lazily from a public repo and however when I tag, push them to a public repo(say quay.io), the lazy image pull effect is not observed.

Following is the log snippet that has some of the error logs collected from the stargz service

..........
Apr 17 09:18:22 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"info","msg":"Received status code: 401 UNAUTHORIZED. Refreshing creds...","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:93871e503e5d40151edd1057993a543bda0fa35937192a7e9de4104df37b7627","time":"2024-04-17T09:18:22.676680752Z"}
Apr 17 09:18:22 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"info","msg":"Received status code: 401 UNAUTHORIZED. Refreshing creds...","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:476de2241a71380a6e9ba94e28e6c532dc42205815937111ec3127104fe65b10","time":"2024-04-17T09:18:22.686206281Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"error":"failed to resolve layer \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\" / \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"debug","msg":"failed to resolve layer","time":"2024-04-17T09:18:23.489656712Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"error":"failed to resolve layer: failed to resolve layer \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\" / \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","layerdigest":"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16","level":"debug","msg":"error resolving layer (context error: \u003cnil\u003e)","remote-snapshot-prepared":"false","time":"2024-04-17T09:18:23.489745500Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"error":"failed to resolve layer: failed to resolve layer \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\" / \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\"","time":"2024-04-17T09:18:23.489786958Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"debug","msg":"reusing manifest and config of \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\"","time":"2024-04-17T09:18:23.493605228Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"debug","msg":"resolving","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:593237bf049fec6b90aaad6ee3638f51f438872a50a17fe9a05dd27d53cca9cf","time":"2024-04-17T09:18:23.493771980Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"debug","msg":"resolving","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:4ef55b3a7bb75e3e6be5c864d74087ea573a5c2c180701ec6292fedea2da183e","time":"2024-04-17T09:18:23.494002151Z"}
..........

I'm using cri-o as the container manager for my cluster and I also tried the workaround mentioned by @dmosdallas but didn't work.

@ktock do you have any leads on this?

dmosdallas mentioned this issue Sep 19, 2023

Images pushed to private repository not fetched lazily #1389

Closed

ElenaHenderson mentioned this issue Apr 24, 2024

fs: Check connection only when image isn't fully cached #1584

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lazy pull from private registry #1107

lazy pull from private registry #1107

reisman234 commented Feb 13, 2023

Dadavan commented Mar 19, 2023 •

edited

ktock commented Mar 20, 2023

Dadavan commented Mar 20, 2023

ktock commented Mar 21, 2023

Dadavan commented Mar 30, 2023

hagaibarel commented Sep 14, 2023

dmosdallas commented Sep 20, 2023

sairameshv commented Apr 17, 2024 •

edited

lazy pull from private registry #1107

lazy pull from private registry #1107

Comments

reisman234 commented Feb 13, 2023

Dadavan commented Mar 19, 2023 • edited

ktock commented Mar 20, 2023

Dadavan commented Mar 20, 2023

ktock commented Mar 21, 2023

Dadavan commented Mar 30, 2023

hagaibarel commented Sep 14, 2023

dmosdallas commented Sep 20, 2023

sairameshv commented Apr 17, 2024 • edited

Dadavan commented Mar 19, 2023 •

edited

sairameshv commented Apr 17, 2024 •

edited