File watching doesn't work on overlay filesystems

[staticfloat]

Hello there! I've run into a curious situation; file watching doesn't seem to work when run inside of a docker container that uses the new overlay or overlay2 storage drivers. To demonstrate this, we just need to start up a recent docker daemon with the overlay or overlay2 storage driver (e.g. sudo dockerd -s overlay2) and use the following Dockerfile:

FROM debian:jessie

RUN apt update; apt install -y autoconf automake libtool git gcc make

RUN git clone -b v1.x https://github.com/libuv/libuv /src/libuv
WORKDIR /src/libuv
RUN sh autogen.sh
RUN ./configure
RUN make -j8
RUN make -j8 check

Building this Dockerfile fails on two tests, namely fs_event_watch_file and fs_event_watch_file_current_dir, both of which timeout with no further output. Interestingly, tools such as inotifywait still work, which I test with the following Dockerfile:

FROM debian:jessie

RUN apt update; apt install -y inotify-tools
RUN echo "#!/bin/bash" > /entrypoint.sh; \
    echo "echo Starting inotify..." >> /entrypoint.sh; \
    echo "inotifywait -m /tmp &" >> /entrypoint.sh; \
    echo "sleep 1" >> /entrypoint.sh; \
    echo "echo Modifying /tmp/a..." >> /entrypoint.sh; \
    echo "touch /tmp/a" >> /entrypoint.sh; \
    echo "sleep 1" >> /entrypoint.sh;

RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

Running the resultant docker image, no matter the storage driver, results in a nice output such as the following:

$ docker run f4f6ddcab821
Starting inotify...
Setting up watches.
Watches established.
Modifying /tmp/a...
/tmp/ CREATE a
/tmp/ OPEN a
/tmp/ ATTRIB a
/tmp/ CLOSE_WRITE,CLOSE a

I am running these tests on an Ubuntu 16.04 host with Docker v1.13 installed, running a 4.4.0 kernel. Using a storage driver of aufs works in all cases.

[bnoordhuis]

Can you check what strace ./run-tests fs_event_watch_file fs_event_watch_file prints vs. strace inotifywait -m /tmp?

If I had to venture a guess, it'd be that there are specific inotify event masks that get on the good side of overlayfs. For example, inotifywait listens for IN_OPEN, libuv does not.

[santigimeno]

It looks inotify might not be fully supported in overlayfs: moby/moby#11705

[staticfloat]

@bnoordhuis here you go. The Dockerfile to build the test environment is as follows:

$ cat Dockerfile
FROM debian:jessie

RUN apt update; apt install -y autoconf automake libtool git gcc make strace inotify-tools

RUN git clone -b v1.x https://github.com/libuv/libuv /src/libuv
WORKDIR /src/libuv
RUN sh autogen.sh
RUN ./configure
RUN make -j8
RUN make -j8 test/run-tests
$ docker build -t strace_test .

When I run strace on the fs_event_watch_file test:

$ docker run --security-opt seccomp:unconfined -ti strace_test strace test/run-tests fs_event_watch_file

I get this output.

When I run strace on inotifywatch while I generate some kind of filesystem activity:

$ docker run --security-opt seccomp:unconfined -ti strace_test /bin/bash -c '(strace inotifywait -m /tmp &); sleep 1; touch /tmp/a; sleep 1;'

I get this output.

[bnoordhuis]

Sorry, can I ask you to run strace -f test/run-tests fs_event_watch_file or run test/.libs/run-tests directly? I forgot that test/run-tests is a shell script when building with autotools.

[staticfloat]

Well, we certainly get quite a bit more information with -f, that's for sure. When I tried to run strace test/.libs/run-tests fs_event_watch_file directly, it complained that it could not find libuv.so.1, and rather than try to dig through the shell script to figure out how it was feeding in libuv.so.1, I decided to just continue to use the shell script, so sorry for the extra noise.

[bnoordhuis]

Thanks. Libuv's event mask: IN_MODIFY|IN_ATTRIB|IN_MOVE|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF

inotifywatch's event mask: IN_ACCESS|IN_MODIFY|IN_ATTRIB|IN_CLOSE|IN_OPEN|IN_MOVE|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF

I suspect it's either the event mask or maybe epoll doesn't work for inotify overlayfs descriptors because the epoll_wait system call never reports events. inotifywatch uses select() and does receive events so it's probably the event mask. What happens when you apply this patch?

diff --git a/src/unix/linux-inotify.c b/src/unix/linux-inotify.c
index 4708c05..8767f4d 100644
--- a/src/unix/linux-inotify.c
+++ b/src/unix/linux-inotify.c
@@ -218,16 +218,18 @@ int uv_fs_event_start(uv_fs_event_t* handle,
   if (uv__is_active(handle))
     return -EINVAL;
 
   err = init_inotify(handle->loop);
   if (err)
     return err;
 
   events = UV__IN_ATTRIB
+         | UV__IN_ACCESS
+         | UV__IN_OPEN
          | UV__IN_CREATE
          | UV__IN_MODIFY
          | UV__IN_DELETE
          | UV__IN_DELETE_SELF
          | UV__IN_MOVE_SELF
          | UV__IN_MOVED_FROM
          | UV__IN_MOVED_TO;
 

[staticfloat]

Nope, it still times out, even with that patch. :(

[bnoordhuis]

Okay, thanks. I think we'll have to chalk this up to bugs in the underlying platform. It doesn't look like there are easy workarounds in libuv.

[staticfloat]

In my testing, this is fixed on kernel v4.10.

[bnoordhuis]

Thanks for following up, I'll close out the issue. Good to hear it works with a newer kernel.