Production-ready process monitoring

[shykes]

For docker to be fully usable in production, we need a robust and standard way to do the following:

1. Auto-restart some long-running processes if they crash
2. Auto-start some processes at system startup
3. Send arbitrary signals to processes

This could be implemented in different ways:

• Option 1 is to implement these features directly in docker, and monitor docker with the host's init process. This is discussed in Auto-restart processes #26 and allow custom signals #230.
• Option 2 is to modify the docker client to behave as a "proxy" to the process it has remotely started, and have the host's init process monitor each instance of the docker client independently. This is the focus of the "dockrun" script by @Backjlack, which he is currently working to merge into docker: Foreground containers should die gracefully when sent a kill signal #507, add support for container ID files (a la pidfile) #1159 and more pull requests to come.
• Option 3 is similar to option 2, except that "docker run" would be modified to be standalone: instead of starting the process remotely via the docker daemon, it would fork/exec itself and monitor its own child process (similarly to chroot). This is discussed in Bring back standalone mode #503.

[shykes]

I would like to find an acceptable solution for the 0.6 release. We don't need to implement all 3. But we need to agree on at least one, and make it work well out-of-the-box.

[unclejack]

#507 is going to be taken care of by #1249. @shykes Is there anything else needed for #1249?

#1249 is going to be the first version of the signal handling passing system. As Guillaume has said, it'd be a good idea to do the same thing for any signal, but that also requires some changes on the API side to allow us to pass an arbitrary signal. SIGINT/SIGTERM should be enough for now, though.

#503 (standalone mode) would be yet another thing to test, so I'd suggest delaying that one until after people see what docker run with features from dockrun can do for them.

[brynary]

Option #1 sounds like it is better handled by existing tools (e.g. supervisord) so that's my least favorite option.

Option #2 is where I lean, but I'm wondering if there are things that will be hard/impossible to do in terms of creating a robust remote proxy for a container. @unclejack do you see any issues towards the eventual goal of having the docker process be effectively transparent with respect to exit codes, signals, and streams?

It seems Option #3 breaks the Docker client/server model.

[unclejack]

@brynary I think it's doable, dockrun was a proof of concept and it worked. So far, docker run has support for container ID files, there's PR #1249 which adds support for handling SIGTERM/SIGINT and support for returning true container exit codes when docker run exits will be in another PR.

The REST API will be extended to send any signal directly to the process running in the container, so we won't be restricted only to SIGTERM/SIGINT in the future.

I'll update dockrun to show how docker run should work with these features.

[dln]

Option #3 does break the client/server model, however it also adds not only signal behavior, but all other aspects of processes as well - Resource monitoring for example. I like my process hierarchy. :)

Separating the concerns of management from actual execution seems more natural IMHO. The executor/"standalone run" could probably register with the docker daemon to still allow management through the remote api, etc, thus preserving the current c/s functionality.

[brynary]

Ah yeah, those benefits make sense. I like the idea of standalone still being registered in the Docker daemon if that doesn't cause other issues.

-Bryan

Bryan Helmkamp, Founder, Code Climate
bryan@brynary.com / 646-379-1810 / @brynary

Sent from my phone. Please forgive brevity and typos^H^H^H^H^Hautocorrect failures.

On Thu, Aug 1, 2013 at 10:20 AM, Daniel Lundin notifications@github.com
wrote:

Option #3 does break the client/server model, however it also adds not only signal behavior, but all other aspects of processes as well - Resource monitoring for example. I like my process hierarchy. :)

Separating the concerns of management from actual execution seems more natural IMHO. The executor/"standalone run" could probably register with the docker daemon to still allow management through the remote api, etc.

Reply to this email directly or view it on GitHub:
#1311 (comment)

[justone]

Would option number 3 mean that the current functionality of spinning up containers and having them be managed by the daemon go away? If so, I would certainly miss it.

Also, if the standalone run just registers itself with the daemon, does that mean I can control it via the daemon? If I can't control it, then there's not much point in registering.

There are two use cases that I can see:

• "I want to use docker as a convenient wrapper around lxc and do the monitoring myself."
• "I want to have docker manage my containers so that I don't have to."

The former would be good for running standalone services or running coordinated services in an environment where you already have a monitoring solution in place. I can see several places that I can use it in this respect. The latter is convenient for small to medium PaaS implementations that don't necessarily want to have their own top level monitoring but rather democratize that into the containers themselves.

Perhaps we could leave docker run as is and create a new subcommand docker exec that behaves just like number 3 above. Then you could choose your level of management at runtime.

[judofyr]

"I want to have docker manage my containers so that I don't have to."

Well, then how much management does Docker do? I've been looking into using Docker in production for some small projects and my main concern now is what happens when the container crashes. Does it restart? Does it notify me by email? There are plenty of process management tools that handle these questions already.

I do see the value of having dockerd spinning up containers as well. Many containers don't require process management (e.g. docker build) and the REST API is nice to have.

Perhaps we could leave docker run as is and create a new subcommand docker exec that behaves just like number 3 above. Then you could choose your level of management at runtime.

+1. docker run would just run docker exec under the dockerd-process.

Is it possible for docker exec to connect to the deamon and register the container there? Then you could control some parts of it (showing status/uptime, sending signals, stopping it). docker logs and docker attach would not work, but that seems fair enough for me.

[brynary]

I'm starting to solidify my thinking on making docker exec the primitive of running a Docker container, and (as @judofyr suggested), building docker run on top of it (managing the process with dockerd).

The exec-primitive is aligned with Unix process management syscalls, and there are worse things to be inspired by. It also should (hopefully) create a nice separation of concerns between run and exec, whereas right now it feels like run is having to be stretched to fit exec-like behavior.

So, relating this back to the original options described by @shykes:

• Option 1 -- Avoid. Too much replicated responsibility in docker.
• Option 2 -- Useful, specifically signals, streams and exit codes, but as has been pointed out here it's still not a real process tree (so you can't e.g. check the resource usage of the proxy process)
• Option 3 -- Yes, in the form of docker exec. For extra credit, we can see about if we can still register these processes into dockerd to get some control of them via REST (and therefore command line clients).

Just my 2 cents.

[justone]

Yeah, I really like having both run and exec. Something @judofyr said triggered a solidification of something in my mind:

...the REST API is nice to have.

It is very useful. The use case that this supports is "I want to manage my containers via the docker daemon and use something other than process level monitoring".

Using docker exec is great for running processes on the same system with a process monitoring tool that runs locally, but in the PaaS scenario opens up other possibilities.

For instance, if I have a PaaS controller node and it talks to 10 remote docker daemons to spin up application containers, I might opt for application level health checks over process level ones (i.e. it doesn't matter if the process is up and running when it's not responding to web requests). In this case, if an application isn't responding, the controller should start the application on another node and stop it on the node that it was running on.

Of course the docker daemon shouldn't be responsible for that level of orchestration, but being able to completely manage containers via the REST API is vital to it being possible.

[ianjw11]

I agree that option 1 posted by @shykes should be avoided, as that sort of functionality is present in many existing applications. I think my ideal use case would be to be able to run a "docker exec" under a supervisord process on the host node.

Not having that functionality, and not being able to easily monitor process trees for usage was the biggest reason why I went with another solution over docker.

[gabrtv]

Big +1 for docker exec.. I thought it was my idea ;)

From what I understand option 2 can achieve the same functionality as option 3, allowing for daemon managers to use docker run in the near-term. Once something like docker exec is implemented, ideally we'd be able to drop it in to a daemon manager without changing any args other than replacing docker run with docker exec.

We'd then benefit from a much shorter call stack, automatic ephemeral behavior (i.e. no container RW layers), little/no daemon interaction or HTTP API calls, natural UNIX process semantics and signal handling, no logging other than to stdout/stderr, etc.

[noteed]

+1 for docker exec. It is a useful/flexible building block, for dockerd itself, but for any other monitoring solution too.

[rafikk]

Has there been any activity on this?

[jpetazzo]

I believe that #2007 implements it! :-)

[mamciek]

It implements only option 2, but what about option 3 (docker exec) ? Any news on that?

[jpetazzo]

I believe that docker exec was superseded by the -sig-proxy option. When the latter is enabled, signals received by the Docker client are propagated to the container. Does that fit the bill?

See: e0b59ab

[denibertovic]

This didn't make into 0.7 I assume? Is there a timeline for when this is expected?
Specifically I'm interested in the "Auto-start some processes at system startup" part.

[gerhard]

Would be really nice to get this working. I'm on 0.7.6, using the -sig-proxy=true option, still no luck:

docker run -a stdout -rm -sig-proxy=true -name=sleeper ubuntu sleep 60

Only SIGKILL gets recognised, and even then, only the docker run process is killed, not the container.

[rubycut]

Confirmed, on 0.7.6 I can't terminate container with term signal.

[rocketraman]

+1 Not working on 0.8.0 either.

[rocketraman]

My preference would be Option #3, but in the meantime I implemented a bash script that wraps docker start and converts a SIGTERM, SIGINT, and SIGQUIT into a docker stop command, and a SIGHUP into a docker restart. It maintains all of the standard output of docker start for running without detaching under a process control manager like supervisord. I believe it is similar in concept to dockrun. Here: https://github.com/rocketraman/docker-infra.

[cpuguy83]

This seems to be completely covered by #7414 using option 1
Option 2 was already implemented.

Option 3 has been discussed but is extremely low priority.

@shykes Can we consider this a closed issue?

[cpuguy83]

Cleaning this one up since the restart policies and docker client signal proxying handle this.

[unclejack]

This should stay open because we don't have single process monitoring (e.g. starting and running a container without the docker cli/daemonless run).