I keep getting an error where buildout seems to be trying to re-install the appdirs egg

[jimfulton]

buildout27
Getting distribution for 'appdirs>=1.4.0'.
zip_safe flag not set; analyzing archive contents...
While:
  Installing.
  Checking for upgrades.
  Getting distribution for 'appdirs>=1.4.0'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/buildout.py", line 1982, in main
    getattr(buildout, command)(args)
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/buildout.py", line 517, in install
    self._maybe_upgrade()
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/buildout.py", line 953, in _maybe_upgrade
    allow_hosts = self._allow_hosts
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/easy_install.py", line 914, in install
    return installer.install(specs, working_set)
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/easy_install.py", line 715, in install
    for dist in self._get_dist(req, ws):
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/easy_install.py", line 564, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/jim/.buildout/eggs/cp27m/zc.buildout-2.9.1-py2.7.egg/zc/buildout/easy_install.py", line 1705, in _move_to_eggs_dir_and_compile
    os.rename(tmp_loc, newloc)
OSError: [Errno 66] Directory not empty

I've seen this several times. Has anyone else seen this? @leorochael does this ring any bells? This started with 2.9.

[jimfulton]

It looks like buildout isn't including appdirs in it's environment for some reason.

[jimfulton]

This is weird. I deleted the old appdirs egg, has to re-bootstrap my shared buildout and now it works fine. :/ I've succeeded flailtastically like this before, until...

[jamadden]

Potentially related? There's recently been a new release of appdirs. I ran into problems installing it yesterday in the gevent build farm because the previous version had invalid metadata and couldn't be uninstalled. In that case the only thing to do was to clear the local virtualenv and start from scratch.

[jimfulton]

See #382 and benjaminp/six#188

At a minimum we should catch this error and display the directory. Perhaps we should even detect the version mismatch.

[jensens]

The following w/o using bootstrap.py works for me in different environments:

A requirements.txt with (needs check if versions are up to date):

appdirs==1.4.2
packaging==16.8
pyparsing==2.1.10
setuptools==34.3.0
six==1.10.0
zc.buildout==2.8.0

And then as shell script or directly on console:

virtualenv --clear .
./bin/pip install -r requirements.txt
./bin/buildout

[jimfulton]

Gaaaa, appdirs has the same bug as six.

[jimfulton]

#382

[leorochael]

ActiveState/appdirs#92

[petri]

Sigh. I am seeing similar os.rename(tmp_loc, newloc) failures in _move_to_eggs_dir_and_compile. But in this case, the error is OSError: [Errno 20] Not a directory. Upon investigating further, there already is an egg file link at newloc.

I've now seen this happen with two different eggs, zc.recipe.egg and collective.recipe.filestorage.

Buildout, I will dance on your grave when the day comes.

[petri]

Ok in my case, this was because of old buildout in which there are checks in _move_to_eggs_dir_and_compile for pre-existing file and directory that cause removal of target first, but there is no a link check, and it looks like the egg file was a link :(

I seem to recall this may have been fixed in newer zc.buildout. On a side note, I wonder why on earth is the zope/ztk kgs still specifying buildout 1.7.1...?

[petri]

Gahhh, the link issue is not fixed in newest 2.9.5 either Bah. The link target (egg) was gone on my system but the link pointing to it was still there. So the os.path.exists check failed, making current buildout assume it's safe to rename. But os.rename fails (as it should) if the destination is an existing link, regardless of whether its target is there or not.

A somewhat rare corner case found, thus.

[leorochael]

On a side note, I wonder why on earth is the zope/ztk kgs still specifying buildout 1.7.1...?

I don't know which kgs specifically you refer to, but in general, packages don't assume they're buildable by more recent builders than the one they were successfully built with.

The link target (egg) was gone on my system but the link pointing to it was still there. So the os.path.exists check failed, making current buildout assume it's safe to rename.

There is only so much sanity check the code can do before attempting to work... It also doesn't check for a read-only filesystem, for example...

The code in question was built with an assumption that separate but simultaneous buildout runs that share an egg directory might race to install the same package at the same time, so they check for the target presence before they move, but don't try to clobber the destination or remove the target before replacing.

The assumption is that whoever put the target there has a good reason to have done so and is probably using it.

It also assumes the "suddenly already existing target" is likely as well built as the newly built one so if it can't move the target into place, it checks the destination is ok and gives up.