logs -f blocks forever when container is stopped

[fsouza]

I don't know whether this is on purpose or not, but we have an issue when using the API for logs. We can't guarantee that the container is not running when we call the /logs endpoints with follow=1, so we may end-up waiting forever.

Is this a bug or the expected behaviour? If the former, I can try to fix it, if the later, I will have to work around it.

[crosbymichael]

Does it start streaming the logs if the container is started and you are still following?

[fsouza]

@crosbymichael when it's started, it behaves as expected: the streaming stops when the container stops.

The issue appears when the container is already stopped in the moment of the /logs call.

[fsouza]

Ok, now I see. It will block waiting for the container to start, but the container won't start again because it has already stopped. This is working as expected. But even when I remove the container it keeps blocked waiting for a stream that will never come, because the container has been removed.

Could we close the streaming connection when the container is removed?

[michaelshobbs]

👍

I use the cli version of this (docker logs -f ) in a dokku plugin and if there is nothing to be done by the plugin, by the time I execute this the container is stopped and thus the deployment will hang.

I actually think it makes sense to support both semantics. Think tail -f vs. tail -F, sort of. 😄

[fsouza]

@michaelshobbs if you always issue the call with the container stopped, you may omit -f.

The problem in tsuru is that we cannot guarantee that we call Logs before the container stops.

[michaelshobbs]

That's the thing, its only the edge case where the container is stopped. Under normal circumstances, the container will be running.

[LK4D4]

@fsouze @michaelshobbs Do you have guys any suggestions how it should be? Just not to begin follow if container is stopped?

[fsouza]

@LK4D4 I believe that would break docker run, not sure. As far I remember, docker run is CreateContainer + AttachToContainer + StartContainer (in order to not lose anything, the command attaches to the container before starting it).

A good approach would be to stop streaming when the container is removed, but I don't know if this would fix the @michaelshobbs case. This is how tsuru approach looks like today:

CreateContainer
StartContainer
AttachToContainer
WaitContainer
CommitContainer
RemoveContainer

AttachToContainer is already broken for the case where the container is stopped, but it hangs inside a goroutine. If Logs cannot replace Attach + Wait call, we aren't able to use it.

[michaelshobbs]

@LK4D4 Let me preface by saying my understanding of the underpinnings of docker are limited. I am primarily a CLI user and have not dug into what is happening at the API level or deeper.

The way docker logs -f functioned before made the most sense to me. If the container is running, print the log from container start (?) and keep the stream open until the container completed.

If the container is stopped, function the same as if -f was not passed. i.e. just print the log, close the stream and exit.

[LK4D4]

@fsouza Really logs now not connected with any other calls, so it can't break anything. If someone for core team approve that we don't stream logs for stopped containers I can implement this.
ping @unclejack @tiborvass @vieux

[tiborvass]

@LK4D4 My opinion is that we should make the behavior consistent, and autoexit logs -f if container is stopped. (Docs and flag help also need update though).
I will let @unclejack or @vieux chime in as well.

[tiborvass]

If we were to introduce -F like @michaelshobbs suggested, it would make a breaking change, since -f should then not quit when the container is running and stops, -F however would stop.

Would this breaking change be that annoying?

[michaelshobbs]

@tiborvass I would actually expect the behavior to be the opposite as -f would be the normal case and quit if the container was stopped and -F would leave the stream open. My expectation of that normal case came from how logs -f worked prior to 1.0. I'm fine with adjusting that if you guys think the opposite makes more sense.

[tiborvass]

@michaelshobbs oh thanks for clarifying. I looked at man tail to doublecheck the behavior. So you're suggesting the opposite of tail for backwards compatibility?

[michaelshobbs]

@tiborvass Backwards compatibility was my thinking, yes.

[tianon]

I'm confused - how would that be the opposite of tail?

With tail -f, if the file disappears, tail finishes. With tail -F, tail will not only follow the file's changes, but will also reopen the file if it is removed and recreated (which I think is a bit much for us, even though it'd be cool to be able to docker logs -F some-container and completely delete "some-container" and recreate a new container with the same name and keep following, but that's pretty heftily complicated and out of scope, I'd think).

So TL;DR I'm +1 for a new docker logs -F ... that doesn't exit if the container stops. :)

[michaelshobbs]

@tianon That's not exactly true. A tail -f will hold onto the file descriptor if the filename is unlinked and will not return unless signaled to quit. However, I still think it makes sense for docker logs -f to behave in the way you described as that is how it behaved previously.

[tianon]

Ah true. +1 :)

[tnachen]

Regardless I think we should let Docker logs --follow exit when the container is completely removed. I'm more or so fine when it's just stopped but if the container is no longer present docker logs --follow shouldn't need to be intervened.

[tnachen]

Found a bug when you launch docker logs --follow after the container stops, pasting the repro steps from #7577

in bash #1
docker run -d busybox echo hi
be83cba526dd07e4af70eadc668c06bf486a68e016acbdf55fb70ad207c0191a
docker logs --follow
hi

after in bash #2
docker rm -f be83cba526dd07e4af70eadc668c06bf486a68e016acbdf55fb70ad207c0191a

looking back at the docker logs --follow process it's still hanging there forever.

[LK4D4]

So, guys, let's decide what we do.
My proposal:
logs -f working as now, but exits immediately if container is stopped.
logs -F exits only on container removal

[michaelshobbs]

👍

[tiborvass]

@LK4D4 I'm fine with that.

[tnachen]

Sounds good, as long the logs process will exit when the container is removed! (which it doesn't now)