Fix error removing diff path #34587

cpuguy83 · 2017-08-21T21:40:40Z

In d42dbdd the code was re-arranged to
better report errors, and ignore non-errors.
In doing so we removed a deferred remove of the AUFS diff path, but did
not replace it with a non-deferred one.

This fixes the issue and makes the code a bit more readable.

tonistiigi · 2017-08-21T22:01:28Z

daemon/graphdriver/aufs/aufs.go

+	// Atomically remove each directory in turn by first moving it out of the
+	// way (so that docker doesn't find it anymore) before doing removal of
+	// the whole tree.
+	if err := atomicRemove(mountpoint); err != nil {


Why is the order of deletion changed in here? I think it makes more sense to remove mount before diffs because they are more likely to get EBUSY(and in some versions, you can't remove diff when it is mounted). And layers last as it should never fail.

I changed this to match the original defer ordering.
I'll double check this later. Fine either way.

I see. This indeed matches the order before #33960

tiborvass · 2017-08-21T22:21:02Z

@cpuguy83 to make this easier for cherrypicking in a patch release, can we make this commit as minimal as possible, addressing the removed defer only?

I suggest you make it "more readable" in another PR, it's important for the diff to be small.

tonistiigi · 2017-08-21T23:21:02Z

Should we add a cleanup of diff/*-removing to aufs.Init() ?

vieux · 2017-08-21T23:41:20Z

@tonistiigi I was about to suggest the same, I think we should for ppl who installed 17.06.1

tiborvass · 2017-08-21T23:55:52Z

@tonistiigi @vieux shouldn't that be only in the patch release, and not master?

vieux · 2017-08-22T00:09:45Z

@tiborvass

if ppl go from 17.06.1 -> 17.07.0 they won't get the fix if we put it in the patch release only

cpuguy83 · 2017-08-22T00:17:28Z

to make this easier for cherrypicking in a patch release, can we make this commit as minimal as possible, addressing the removed defer only?

The primary reason I moved it out rather than keeping it all in one function is I needed to add additional error handling for the IsExist(Err) case where a remove fails, and now we have a dir that has been renamed but not removed, which just added to the complexity.

Should we add a cleanup of diff/*-removing to aufs.Init() ?
SGTM

andrewhsu · 2017-08-22T00:40:41Z

Would it be suitable to have a unit test in the corresponding aufs_test.go?

cpuguy83 · 2017-08-22T12:45:08Z

Added tests to make sure that there's no -removing dir hanging around after Remove() and also that Init() cleans up dirs with -removing

vieux · 2017-08-22T16:07:39Z

@tonistiigi @tiborvass LGTY now? Let's get this one in asap

tonistiigi

LGTM

tonistiigi · 2017-08-22T16:24:07Z

daemon/graphdriver/aufs/aufs_test.go

@@ -149,6 +151,9 @@ func TestRemoveImage(t *testing.T) {
 		if _, err := os.Stat(path.Join(tmp, p, "1")); err == nil {
 			t.Fatalf("Error should not be nil because dirs with id 1 should be delted: %s", p)
 		}
+		if _, err := os.Stat(path.Join(tmp, p, "1-removing")); err == nil {
+			t.Fatalf("Error should not be nil because dirs with id 1-removing should be delted: %s", p)


In d42dbdd the code was re-arranged to better report errors, and ignore non-errors. In doing so we removed a deferred remove of the AUFS diff path, but did not replace it with a non-deferred one. This fixes the issue and makes the code a bit more readable. Signed-off-by: Brian Goff <cpuguy83@gmail.com>

vieux · 2017-08-22T16:53:43Z

LGTM

tiborvass · 2017-08-22T16:56:02Z

daemon/graphdriver/aufs/aufs.go

+		p := filepath.Join(root, path)
+		dirs, err := ioutil.ReadDir(p)
+		if err != nil {
+			logrus.WithError(err).WithField("dir", p).Error("error reading dir entries")


Would be great to mention aufs in the error message.

Unless some higher caller already inserts that info

tiborvass · 2017-08-22T16:57:37Z

daemon/graphdriver/aufs/aufs.go

@@ -142,6 +142,23 @@ func Init(root string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
 		}
 	}

+	for _, path := range []string{"mnt", "diff"} {
+		p := filepath.Join(root, path)
+		dirs, err := ioutil.ReadDir(p)


It is a bit confusing to call the variable dirs because they may not all be dirs. Entries?

tiborvass · 2017-08-22T16:58:39Z

daemon/graphdriver/aufs/aufs.go

+		}
+		for _, dir := range dirs {
+			if strings.HasSuffix(dir.Name(), "-removing") {
+				logrus.WithField("dir", dir.Name()).Debug("Cleaning up stale layer dir")


Mention aufs

tiborvass · 2017-08-22T16:59:11Z

daemon/graphdriver/aufs/aufs.go

+			if strings.HasSuffix(dir.Name(), "-removing") {
+				logrus.WithField("dir", dir.Name()).Debug("Cleaning up stale layer dir")
+				if err := system.EnsureRemoveAll(filepath.Join(p, dir.Name())); err != nil {
+					logrus.WithField("dir", dir.Name()).WithError(err).Error("Error removing stale layer dir")


tiborvass · 2017-08-22T17:11:44Z

daemon/graphdriver/aufs/aufs.go

@@ -317,31 +334,25 @@ func (a *Driver) Remove(id string) error {
 		retries++
 		logger.Warnf("unmount failed due to EBUSY: retry count: %d", retries)
 		time.Sleep(100 * time.Millisecond)
-		continue


Sorry I can't see on the phone the whole function. Just want to make sure the return statement at the end the block is wanted and shouldn't instead be a logging statement followed by this removed continue.

it's a continue at the end of a for loop, it was useless

andrewhsu · 2017-08-22T17:22:16Z

Test LGTM: I've verified that without this fix, the tests added to aufs_test.go fail on the master branch:

$ make test-unit
...
--- FAIL: TestRemoveImage (0.00s)
	aufs_test.go:155: Error should not be nil because dirs with id 1-removing should be delted: diff
--- FAIL: TestInitStaleCleanup (0.02s)
	aufs_test.go:824: cleanup failed
FAIL
...

Nit-pick (not required for merge): in the future, it may be more helpful to have changes to tests as a separate git commit to the PR. Also, spelling: delted.

tiborvass · 2017-08-22T17:51:41Z

@andrewhsu the spelling mistake was fixed already.

LGTM despite minor nitpicks

vieux · 2017-08-22T18:02:37Z

let's do a follow up PR

justlaputa · 2017-09-06T23:08:16Z

@cpuguy83 hi, do you have plan to backport this fix to older version of docker, like 1.10 or 1.12?

cpuguy83 · 2017-09-06T23:16:34Z

@judtlaputa this is a specific fix for a regression in 17.06.1.

justlaputa · 2017-09-06T23:32:26Z

@cpuguy83 oh, sorry. I should comment on the original PR #33960

resolves annoying disk space eater "-removing" regression with aufs/diff moby/moby#34587 moby/moby#21925 (comment) https://stackoverflow.com/a/45798794 moby/moby#22207

GordonTheTurtle added the status/0-triage label Aug 21, 2017

cpuguy83 added status/2-code-review and removed status/0-triage labels Aug 21, 2017

cpuguy83 mentioned this pull request Aug 21, 2017

orphaned diffs #22207

Closed

tonistiigi reviewed Aug 21, 2017

View reviewed changes

cpuguy83 force-pushed the aufs_rm_errors branch 2 times, most recently from 817aeb2 to 84ba209 Compare August 22, 2017 00:57

vieux added the rebuild/experimental label Aug 22, 2017

GordonTheTurtle removed the rebuild/experimental label Aug 22, 2017

tonistiigi approved these changes Aug 22, 2017

View reviewed changes

cpuguy83 force-pushed the aufs_rm_errors branch from 84ba209 to 276b446 Compare August 22, 2017 16:52

tiborvass reviewed Aug 22, 2017

View reviewed changes

vieux merged commit bbcdc7d into moby:master Aug 22, 2017

cpuguy83 deleted the aufs_rm_errors branch August 22, 2017 18:12

This was referenced Aug 22, 2017

[17.06] backport fix for removing diff path docker/docker-ce#201

Merged

[17.07] backport fix for removing diff path docker/docker-ce#202

Merged

pld-gitsync pushed a commit to pld-linux/docker-ce that referenced this pull request Sep 9, 2017

up to 17.06.2

7a8d5be

resolves annoying disk space eater "-removing" regression with aufs/diff moby/moby#34587 moby/moby#21925 (comment) https://stackoverflow.com/a/45798794 moby/moby#22207

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix error removing diff path #34587

Fix error removing diff path #34587

cpuguy83 commented Aug 21, 2017

tonistiigi Aug 21, 2017

cpuguy83 Aug 21, 2017

tonistiigi Aug 21, 2017

tiborvass commented Aug 21, 2017

tonistiigi commented Aug 21, 2017

vieux commented Aug 21, 2017

tiborvass commented Aug 21, 2017

vieux commented Aug 22, 2017

cpuguy83 commented Aug 22, 2017

andrewhsu commented Aug 22, 2017

cpuguy83 commented Aug 22, 2017

vieux commented Aug 22, 2017

tonistiigi left a comment

tonistiigi Aug 22, 2017

vieux commented Aug 22, 2017

tiborvass Aug 22, 2017

tiborvass Aug 22, 2017

tiborvass Aug 22, 2017

tiborvass Aug 22, 2017

tiborvass Aug 22, 2017

tiborvass Aug 22, 2017

vieux Aug 22, 2017

andrewhsu commented Aug 22, 2017 •

edited

tiborvass commented Aug 22, 2017

vieux commented Aug 22, 2017 •

edited

justlaputa commented Sep 6, 2017

cpuguy83 commented Sep 6, 2017

justlaputa commented Sep 6, 2017

Fix error removing diff path #34587

Fix error removing diff path #34587

Conversation

cpuguy83 commented Aug 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiborvass commented Aug 21, 2017

tonistiigi commented Aug 21, 2017

vieux commented Aug 21, 2017

tiborvass commented Aug 21, 2017

vieux commented Aug 22, 2017

cpuguy83 commented Aug 22, 2017

andrewhsu commented Aug 22, 2017

cpuguy83 commented Aug 22, 2017

vieux commented Aug 22, 2017

tonistiigi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vieux commented Aug 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewhsu commented Aug 22, 2017 • edited

tiborvass commented Aug 22, 2017

vieux commented Aug 22, 2017 • edited

justlaputa commented Sep 6, 2017

cpuguy83 commented Sep 6, 2017

justlaputa commented Sep 6, 2017

andrewhsu commented Aug 22, 2017 •

edited

vieux commented Aug 22, 2017 •

edited