[DO NOT MERGE] daemon.ContainerLogs(): fix --follow

[kolyshkin]

This is a continuation of #37576, much simplified and improved, also easier to review

This commit tires to kill three birds with one stone.

I. Goroutine leak on docker logs --follow (#37391).

When daemon.ContainerLogs() is called with options.follow=true
(as in "docker logs --follow"), the "loggerutils.followLogs()"
function never returns (even then the logs consumer is gone).
As a result, all the resources associated with it (including
an opened file descriptor for the log file being read, two FDs
for a pipe, and two FDs for inotify watch) are never released.

If this is repeated (such as by running "docker logs --follow"
and pressing Ctrl-C a few times), this results in DoS caused by
either hitting the limit of inotify watches, or the limit of
opened files. The only cure is daemon restart.

Apparently, what happens is:

1. logs producer (a container) is gone, calling (*LogWatcher).Close()
   for all its readers (daemon/logger/jsonfilelog/jsonfilelog.go:175).

2. WatchClose() is properly handled by a dedicated goroutine in
   followLogs(), cancelling the context.

3. Upon receiving the ctx.Done(), the code in followLogs()
   (daemon/logger/loggerutils/logfile.go#L626-L638) keeps to
   send messages synchronously (which is OK for now).

4. Logs consumer is gone (Ctrl-C is pressed on a terminal running
   "docker logs --follow"). Method (*LogWatcher).Close() is properly
   called (see daemon/logs.go:114). Since it was called before and
   due to to once.Do(), nothing happens (which is kinda good, as
   otherwise it will panic on closing a closed channel).

5. A goroutine (see item 3 above) keeps sending log messages
   synchronously to the logWatcher.Msg channel. Since the
   channel reader is gone, the channge send operation blocks forever,
   and resource cleanup set up in defer statements at the beginning
   of followLogs() never happens.

II. Premature exit of docker logs --follow (#37630).

docker logs --follow should mimic tail -f, which only exits once
the file it watches is gone. Same here: even if a container is
stopped, docker logs --follow should not exit since the container
can be restarted and continue producing logs. The two ways to
exit "docker logs --follow" should be:

1. Kill it (as in Ctrl-C or kill).

2. Remove the container.

III. Complicated logic of following logs.

This is not an issue per se, but followLogs() is "pretty gnarly"
(C) @cpuguy83. While we're at it, let's try to improve things a bit.

Now onto the fix.

1. It appears to be that LogWatcher should not be bothered with
   log producer (i.e. container) being gone, so remove calls to
   (*LogWatcher).Close() from the log drivers, as well as the
   accompaniying data structures. This helps to solve issues II and III.

2. To clarify what (*LogWatcher).Close() actually is, rename it
   to ConsumerGone() (note before that patch it used to mean both
   ProducerGone() (which is eliminated in 1 above) and ConsumerGone().
   Similarly, WatchClose() is now WatchConsumerGone(), etc.
   This is not required, but improves code readability a bit.

3. followLogs() is modified to:

• remove the context and the associated goroutine, which is no longer
  needed (helps to fix II and III);
• remove blocking msg send (fixes I);
• watch for Chmod event (which, if the file being watched is no more,
  means that the (opened) file is removed), exiting if the log file
  is removed (helps to fix II);
• exit once ConsumerGone() is received, freeing all the
  resources (fixes I).

4. Some existing test cases are modified due to II being fixed (i.e. now the container
   should be removed if we want docker logs -f to finish by itself).

5. A test case TestLogsFollowGoroutineLeak is added to check if I is fixed.

6. ContainerLogs() and ReadLogs() are modified to honor "follow" flag
   even if the container is not running.

[thaJeztah]

Per fsnotify/fsnotify#183 (comment) and fsnotify/fsnotify#245 (comment) the project is looking for new maintainers; perhaps we can assist in that

[kolyshkin]

I'll start with a few PRs :)

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@0d9d861). Click here to learn what that means.
The diff coverage is 23.33%.

@@            Coverage Diff            @@
##             master   #37656   +/-   ##
=========================================
  Coverage          ?   36.04%           
=========================================
  Files             ?      609           
  Lines             ?    45018           
  Branches          ?        0           
=========================================
  Hits              ?    16225           
  Misses            ?    26565           
  Partials          ?     2228

[kolyshkin]

Windows CI failure looks legit:

02:07:42.875 FAIL: check_test.go:107: DockerSuite.TearDownTest
02:07:42.875
02:07:42.875 assertion failed: error is not nil: Error response from daemon: unable to remove filesystem for 8f40ac43ca11be91cd139bf4b75db1164ed0639140757d3c9589b3bdda5e1ba2: remove D:\CI\CI-0f0bc06c5\daemon\containers\8f40ac43ca11be91cd139bf4b75db1164ed0639140757d3c9589b3bdda5e1ba2\8f40ac43ca11be91cd139bf4b75db1164ed0639140757d3c9589b3bdda5e1ba2-json.log: The process cannot access the file because it is being used by another process.: failed to remove 8f40ac43ca11be91cd139bf4b75db1164ed0639140757d3c9589b3bdda5e1ba2

[kolyshkin]

docker.py test case hang is also legit; made a couple of fixes to tests: docker/docker-py#2121; will retest.

[kolyshkin]

z ci needs to be restarted

[anusha-ragunathan]

Merge conflict needs to be resolved

[kolyshkin]

Rebased; local log driver patched as well (to remove readers).

[kolyshkin]

rebased; should help CI

[kolyshkin]

As much as I like this PR (it is relatively simple and elegant, and removes a lot of code), I am afraid it won't work, for the following reasons:

• This PR makes a backward-incompatible change. It seems that existing software relies on current behavior of ContainerLogs(follow=true) to finish once a container is stopped (in other words, docker logs -f exits whenever container stops #37630 is not a bug but a feature). For one example of how this is backward-incompatible, see Fixes to logs --follow docker/docker-py#2121.
• This PR relies on a feature to send remove event for an opened file which is currently not supported by fsnotify (Do not suppress Chmod on non-existent file fsnotify/fsnotify#260, Missing Remove Event when file handle open before for Linux fsnotify/fsnotify#194);
• The above feature can't possibly work on Windows (as far as I know, one can't remove an opened file in Windows).

With that said, I see no other way but to return to #37576 -- which is complicated and ugly, but backward-compatible and solves the initial issue (#37391).