No support inotify_add_watch() in docker overlayfs storage backend

[brain-zhang]

Hi, I try to start a container which use overlayfs.

when i exec:

tail -f xx.log

it'll block after show serveral headlines.

i exec:

strace tail -n1 -f xx.log

it shows:

...
inotify_init()                          = 4
inotify_add_watch(4, "test.txt", IN_MODIFY|IN_ATTRIB|IN_DELETE_SELF|IN_MOVE_SELF) = 1
fstat(3, {st_mode=S_IFREG|0644, st_size=5470, ...}) = 0
read(4,

then block

i exec:

inotifywatch -v -e access -e modify -t 5 -r xx.log

show:

No events occurred.

but i has sustained flushing text to xx.log

so why doesn't overlayfs support inotify_add_watch()?

sysinfo:

uanme -a

Linux 3b6656e0c11c 3.19.0-9-generic #9-Ubuntu SMP Wed Mar 11 17:50:03 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker -v

Docker version 1.5.0-dev, build a78ce5c

[dqminh]

IIRC, overlayfs does not fully support inotify ( so tail / inotifywatch may not work )

[brain-zhang]

Hi,I found history discussion here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/882147

It seems that --disable-inotify is a good solution and merits being made into a feature.

So i exec:

tail -f ---disable-inotify xx.log

it works.

[bobrik]

http://whichstoragedrivershouldiuse.github.io/

Easy, the one without inotify support :(

[wuxxin]

for testing, you could try: https://launchpadlibrarian.net/149319334/overlayfs_inotify.patch

[purew]

Ran into this problem today with docker 1.8.2 using overlay-fs. Testcase:

docker run -it ubuntu
apt-get update && apt-get install -y inotify-tools
touch /tmp/test && inotifywait -e delete_self /tmp/test & sleep 1 && rm /tmp/test

Correct output should be:

Setting up watches.
Watches established.
/tmp/test DELETE_SELF

For some reason, IN_ATTRIB is still raised so my temporary workaround is to react to that signal instead for now.

[philips]

@wuxxin Is anyone pushing this patch upstream?

cc @mjg59

[nehaljwani]

Another example where it works when the storage driver is devicemapper, but doesn't work as expected when the storage driver is overlayfs:

#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>
#include <stdio.h>
#include <assert.h>
#include <sys/inotify.h>
#include <sys/syscall.h>

struct InotifyEvent {
  int      wd;
  uint32_t mask;
  uint32_t cookie;
  uint32_t len;
  char     name __flexarr;
};

int main() {
    int fd;
    int inotifyFd;
    int watchDescriptor;
    char file[] = "/tmp/inotify_test.XXXXXX";
    struct InotifyEvent* event;

    printf("Inotify on a File.\n");

    // Create a file
    fd = mkstemp(file);
    assert(fd > -1);

    inotifyFd = inotify_init();
    assert(inotifyFd > -1);

    // add watch on the file for modify and delete
    watchDescriptor = inotify_add_watch(inotifyFd,
                       file,
                       IN_MODIFY |
                                       IN_DELETE_SELF);
    assert(watchDescriptor > -1);

    // modify file;
    write(fd, "test", sizeof("test"));
    fflush(NULL);

    // check modify
    event = reinterpret_cast<InotifyEvent*>(malloc(sizeof(*event)));

    assert(read(inotifyFd, event, sizeof(InotifyEvent)) > 0);

    assert(watchDescriptor == event->wd);
    assert(IN_MODIFY == event->mask);

    // check delete
    assert(0 == remove(file));
    assert(0 == close(fd));

    assert(read(inotifyFd, event, sizeof(InotifyEvent)) > 0);

    assert(watchDescriptor == event->wd);
    assert(IN_DELETE_SELF == event->mask);
    free(event);
}

In overlayfs, the first read() gets stuck. Another discussion which didn't have any follow up: http://lkml.iu.edu/hypermail/linux/kernel/1202.3/01782.html

[sam-thibault]

This is an old issue. I will close it as stale.