List API includes partially-created containers which cannot be inspected #44512
Labels
area/api
kind/bug
Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed.
status/confirmed
Comments
belugabehr
added
kind/bug
|
Are you able to reproduce the issue with a more recent version? This issue might have been fixed by #43166 in v20.10.13. |
|
One correction I'll make is that the title of this report is "Cannot Inspect" but in my scenario, the requests for both inspect and delete returned 404 error codes. |
|
One other observation, based on some weird re-try code in our product to work around this issue, this issue is happening with our currently deployed version of docker-1.12.6. For whatever reason, we are seeing it occur more regularly when upgrading to 20.10.7 |
|
I reproduced the issue on the latest master after widening the race window. Before the Create API request returns, the List endpoint returns the container. Attempting to inspect a not-quite-created container doesn't just return 404: the daemon nil-dereference panics! Yikes. (Note: this is not a DoS or other security vulnerability. The daemon does not crash and remains responsive to subsequent API requests.) diff --git a/daemon/create.go b/daemon/create.go
index b9efba80f7..9f3418055c 100644
--- a/daemon/create.go
+++ b/daemon/create.go
@@ -193,6 +193,10 @@ func (daemon *Daemon) create(ctx context.Context, opts createOpts) (retC *contai
return nil, err
}
+ logrus.WithField("id", ctr.ID).Info("Sleeping 30s in the middle of creating container")
+ time.Sleep(30 * time.Second)
+ logrus.WithField("id", ctr.ID).Info("Done sleeping")
+
var endpointsConfigs map[string]*networktypes.EndpointSettings
if opts.params.NetworkingConfig != nil {
endpointsConfigs = opts.params.NetworkingConfig.EndpointsConfigroot@c6cf37d0b0fe:/go/src/github.com/docker/docker# docker create hello-world &
[2] 9580
root@c6cf37d0b0fe:/go/src/github.com/docker/docker# Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:faa03e786c97f07ef34423fccceeec2398ec8a5759259f94d99078f264e9d7af
Status: Downloaded newer image for hello-world:latest
INFO[2022-12-09T19:36:33.953695745Z] Sleeping 30s in the middle of creating container id=9a3ae6f15fbb4a85a66511356c46e2db325adc2f95ed321eafe3a26957716a88
root@c6cf37d0b0fe:/go/src/github.com/docker/docker# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9a3ae6f15fbb hello-world "/hello" 6 seconds ago Created agitated_mcclintock
root@c6cf37d0b0fe:/go/src/github.com/docker/docker# docker inspect 9a3ae6f15fbb4a85a66511356c46e2db325adc2f95ed321eafe3a26957716a88
2022/12/09 19:36:44 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 298 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1850 +0xbf
panic({0x563f153c4e80, 0x563f169dd520})
/usr/local/go/src/runtime/panic.go:890 +0x262
github.com/docker/docker/daemon.(*Daemon).ContainerInspectCurrent(0xc00086ec80, {0xc00098e166?, 0x563f14c91f50?}, 0x0)
/go/src/github.com/docker/docker/daemon/inspect.go:40 +0x54
github.com/docker/docker/daemon.(*Daemon).ContainerInspect(0x563f1539c2c0?, {0xc00098e166, 0x40}, 0x4?, {0xc00098e156, 0x4})
/go/src/github.com/docker/docker/daemon/inspect.go:29 +0xbe
github.com/docker/docker/api/server/router/container.(*containerRouter).getContainersByName(0xc000f3de00, {0x563f15760fa0, 0xc00130c7b0}, {0x563f1575f4f0, 0xc0014b20e0}, 0x563f152b1d20?, 0xc000c64dc0?)
/go/src/github.com/docker/docker/api/server/router/container/inspect.go:15 +0xba
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0x563f15760fa0, 0xc00130c7b0}, {0x563f1575f4f0?, 0xc0014b20e0?}, 0x563f1529afe0?, 0xc0007d2500?)
/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x15b
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0x563f15760fa0, 0xc00130c780}, {0x563f1575f4f0, 0xc0014b20e0}, 0x203000?, 0x40?)
/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x4d7
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0x563f15760fa0, 0xc00130c780}, {0x563f1575f4f0?, 0xc0014b20e0}, 0xc000895000, 0xc0007d24a0?)
/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x649
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0x563f1575f4f0, 0xc0014b20e0}, 0xc000894f00)
/go/src/github.com/docker/docker/api/server/server.go:144 +0x1ce
net/http.HandlerFunc.ServeHTTP(0xc000894d00?, {0x563f1575f4f0?, 0xc0014b20e0?}, 0xc000cad9e0?)
/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc000c0d500, {0x563f1575f4f0, 0xc0014b20e0}, 0xc000927200)
/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xc001250f00?}, {0x563f1575f4f0, 0xc0014b20e0}, 0xc000927200)
/usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc00016c0a0, {0x563f15760fa0, 0xc0010fd080})
/usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3102 +0x4db
2022/12/09 19:36:44 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 360 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1850 +0xbf
panic({0x563f153c4e80, 0x563f169dd520})
/usr/local/go/src/runtime/panic.go:890 +0x262
github.com/docker/docker/daemon.(*Daemon).ContainerInspectCurrent(0xc00086ec80, {0xc00098e246?, 0x563f14c91f50?}, 0x0)
/go/src/github.com/docker/docker/daemon/inspect.go:40 +0x54
github.com/docker/docker/daemon.(*Daemon).ContainerInspect(0x563f1539c2c0?, {0xc00098e246, 0x40}, 0x4?, {0xc00098e236, 0x4})
/go/src/github.com/docker/docker/daemon/inspect.go:29 +0xbe
github.com/docker/docker/api/server/router/container.(*containerRouter).getContainersByName(0xc000f3de00, {0x563f15760fa0, 0xc00130ce10}, {0x563f1575f4f0, 0xc0014b22a0}, 0x563f152b1d20?, 0xc000c651e0?)
/go/src/github.com/docker/docker/api/server/router/container/inspect.go:15 +0xba
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1({0x563f15760fa0, 0xc00130ce10}, {0x563f1575f4f0?, 0xc0014b22a0?}, 0x563f1529afe0?, 0xc0007d2690?)
/go/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x15b
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1({0x563f15760fa0, 0xc00130cde0}, {0x563f1575f4f0, 0xc0014b22a0}, 0x203000?, 0x40?)
/go/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x4d7
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1({0x563f15760fa0, 0xc00130cde0}, {0x563f1575f4f0?, 0xc0014b22a0}, 0xc000895400, 0xc0007d2640?)
/go/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x649
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1({0x563f1575f4f0, 0xc0014b22a0}, 0xc000895300)
/go/src/github.com/docker/docker/api/server/server.go:144 +0x1ce
net/http.HandlerFunc.ServeHTTP(0xc000895200?, {0x563f1575f4f0?, 0xc0014b22a0?}, 0x800?)
/usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc000c0d500, {0x563f1575f4f0, 0xc0014b22a0}, 0xc000895100)
/go/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xc00130ca50?}, {0x563f1575f4f0, 0xc0014b22a0}, 0xc000895100)
/usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc000ee41e0, {0x563f15760fa0, 0xc0010fd080})
/usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3102 +0x4db
[]
error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.30/containers/9a3ae6f15fbb4a85a66511356c46e2db325adc2f95ed321eafe3a26957716a88/json: EOF |
corhere
changed the title
corhere
added a commit
to corhere/moby
that referenced
this issue
Dec 12, 2022
(*Container).CheckpointTo() upserts a snapshot of the container to the daemon's in-memory ViewDB and also persists the snapshot to disk. It does not register the live container object with the daemon's container store, however. The ViewDB and container store are used as the source of truth for different operations, so having a container registered in one but not the other can result in inconsistencies. In particular, the List Containers API uses the ViewDB as its source of truth and the Container Inspect API uses the container store. The (*Daemon).setHostConfig() method is called fairly early in the process of creating a container, long before the container is registered in the daemon's container store. Due to a rogue CheckpointTo() call inside setHostConfig(), there is a window of time where a container can be included in a List Containers API response but "not exist" according to the Container Inspect API and similar endpoints which operate on a particular container. Remove the rogue call so that the caller has full control over when the container is checkpointed and update callers to checkpoint explicitly. No changes to (*Daemon).create() are needed as it checkpoints the fully-created container via (*Daemon).Register(). Fixes moby#44512. Signed-off-by: Cory Snider <csnider@mirantis.com>
corhere
added a commit
to corhere/moby
that referenced
this issue
Dec 13, 2022
(*Container).CheckpointTo() upserts a snapshot of the container to the daemon's in-memory ViewDB and also persists the snapshot to disk. It does not register the live container object with the daemon's container store, however. The ViewDB and container store are used as the source of truth for different operations, so having a container registered in one but not the other can result in inconsistencies. In particular, the List Containers API uses the ViewDB as its source of truth and the Container Inspect API uses the container store. The (*Daemon).setHostConfig() method is called fairly early in the process of creating a container, long before the container is registered in the daemon's container store. Due to a rogue CheckpointTo() call inside setHostConfig(), there is a window of time where a container can be included in a List Containers API response but "not exist" according to the Container Inspect API and similar endpoints which operate on a particular container. Remove the rogue call so that the caller has full control over when the container is checkpointed and update callers to checkpoint explicitly. No changes to (*Daemon).create() are needed as it checkpoints the fully-created container via (*Daemon).Register(). Fixes moby#44512. Signed-off-by: Cory Snider <csnider@mirantis.com> (cherry picked from commit 0141c6d) Signed-off-by: Cory Snider <csnider@mirantis.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I am working on an application that was implemented before the various
docker system prunecommands were available and therefore had manually implemented pruning from the client side. The logic is pretty straightforward:This process fails with some frequency as there are times when attempting to inspect a container that was returned by list returns a 404 code. I've noticed that it consistently fails to inspect containers that are in a "created" state.
In the application, there is one thread that is regularly performing this prune process, and another thread that is launching containers by first creating the container and then starting it.
Subsequent attempts to prune the container are successful. It seems perhaps that there is a small window of time where the container is in a state that makes it a candidate for list but not for inspect.
Reproduce
Happens randomly. I was unable to mimic this with CLI.
Expected behavior
Assuming no external modification between calls, all containers returned by
listare available toinspect.docker version
docker info
Additional Info
This is from the client perspective (docker-java):
The text was updated successfully, but these errors were encountered: