Bootstrap script unable to locate releases following 20.10.0 release #557

idgserpro · 2016-04-25T15:23:25Z

How to test:

Download

https://bootstrap.pypa.io/bootstrap-buildout.py

At the time of this ticket, __version__ of this file is 2015-07-01.

Run

python bootstrap-buildout.py

You get

Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-20.10.1.zip
Extracting in /tmp/tmpIKldPD
Traceback (most recent call last):
  File "bootstrap-buildout.py", line 117, in <module>
    ez['use_setuptools'](**setup_args)
  File "<string>", line 173, in use_setuptools
  File "<string>", line 124, in _do_download
  File "<string>", line 64, in _build_egg
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "<string>", line 103, in archive_context
  File "/usr/lib/python2.7/zipfile.py", line 714, in __init__
    self._GetContents()
  File "/usr/lib/python2.7/zipfile.py", line 748, in _GetContents
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 763, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

You get the same error with 20.10.0.

python bootstrap-buildout.py --setuptools-version=20.10.0

Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-20.10.0.zip
Extracting in /tmp/tmp6G6ZVT
Traceback (most recent call last):
  File "bootstrap-buildout.py", line 117, in <module>
    ez['use_setuptools'](**setup_args)
  File "<string>", line 173, in use_setuptools
  File "<string>", line 124, in _do_download
  File "<string>", line 64, in _build_egg
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "<string>", line 103, in archive_context
  File "/usr/lib/python2.7/zipfile.py", line 714, in __init__
    self._GetContents()
  File "/usr/lib/python2.7/zipfile.py", line 748, in _GetContents
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 763, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

20.9.0 works fine, so this bug was introduced in 20.10.x.

python bootstrap-buildout.py --setuptools-version=20.9.0

Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-20.9.0.zip
Extracting in /tmp/tmpXrDIjV
Now working in /tmp/tmpXrDIjV/setuptools-20.9.0
Building a Setuptools egg in /tmp/bootstrap-OVHpkN
warning: no files found matching '*' under directory 'setuptools/_vendor'
warning: no files found matching '*.txt'
/tmp/bootstrap-OVHpkN/setuptools-20.9.0-py2.7.egg

The text was updated successfully, but these errors were encountered:

sbocconi · 2016-04-25T15:47:42Z

The same, breaks Vagrant/Puppet installation:
wget https://bootstrap.pypa.io/ez_setup.py -O /.puphpet-stuff/ez_setup.py && python /.puphpet-stuff/ez_setup.py

--2016-04-25 15:46:44--  https://bootstrap.pypa.io/ez_setup.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 185.31.17.175
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|185.31.17.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12059 (12K) [text/x-python]
Saving to: ‘/.puphpet-stuff/ez_setup.py’

100%[====================================================================================================================================================================================================>] 12,059      --.-K/s   in 0s      

2016-04-25 15:46:44 (77.5 MB/s) - ‘/.puphpet-stuff/ez_setup.py’ saved [12059/12059]

Extracting in /tmp/tmp32iyZh
Traceback (most recent call last):
  File "/.puphpet-stuff/ez_setup.py", line 415, in <module>
    sys.exit(main())
  File "/.puphpet-stuff/ez_setup.py", line 412, in main
    return _install(archive, _build_install_args(options))
  File "/.puphpet-stuff/ez_setup.py", line 52, in _install
    with archive_context(archive_filename):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/.puphpet-stuff/ez_setup.py", line 103, in archive_context
    with ContextualZipFile(filename) as archive:
  File "/.puphpet-stuff/ez_setup.py", line 87, in __new__
    return zipfile.ZipFile(*args, **kwargs)
  File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 811, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

FrancescoRizzi · 2016-04-25T15:49:14Z

I just hit the same "File is not a zip file" error by simply downloading ez_setup.py from the official recommended source and running it

cyrilgdn · 2016-04-25T16:10:45Z

Seems that the problem is: https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package and ez_setup has the old pypi URL format hard-coded....
(He use https://pypi.python.org/pypi/setuptools/json to get last version but not right URLs :-( )

idgserpro · 2016-04-25T18:55:04Z

@jaraco Do you have any idea if it's possible to add a test case in setuptools for this situation?

idgserpro · 2016-04-25T19:18:17Z

@cyrilgdn Indeed, the xml invalid response from the url comes as a 313 bytes zip file. Try https://pypi.python.org/packages/source/s/setuptools/setuptools-20.10.1.zip and you get

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>source/s/setuptools/setuptools-20.10.1.zip</Key>
<RequestId>DAC65125FC40ADDB</RequestId>
<HostId>NqleAbKasjoaBoOZhowtOhluWuTzEgiZZ0HIbREPxqq9wxn6VbbdT0P5r+okwXAk4lzIq597yFA=</HostId>
</Error>

https://pypi.python.org/packages/source/s/setuptools/setuptools-20.9.0.zip works.

@dstufft if possible, any news for https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package#comment-27245473?

@jaraco would it be the case to add a _resolve_setuptools_url_download like we have _resolve_version in ez_setup.py?

jimfulton · 2016-04-25T19:53:32Z

There's a pull request if someone would review it please.

foursue · 2016-04-25T20:31:20Z

+1

idgserpro · 2016-04-25T20:46:08Z

Can we safely use pypi.io (see the full pull request from @jimfulton #559) in the state it's now, in pre-production? As of now, we have these statistics for setuptools:

Downloads (All Versions):
581 downloads in the last day
1173985 downloads in the last week
5004041 downloads in the last month

Is pypi.io able to handle the load? I'm not into PYPI 2.0 project so I don't have clues about that.

IMHO, I would like to have the fix done for pypi.io in https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package#comment-27243225 in pypi.python.org as well, but don't know if it's possible. @dstufft is it possible/feasible? Is this workaround temporary? If so, for how long?

jimfulton · 2016-04-25T21:19:04Z

Folks, my PR fixes this.

It used pypi.io and it adds the --location option to curl.

dstufft · 2016-04-25T22:50:12Z

Can you post the full output of curl -I -L https://pypi.io/packages/source/s/setuptools/setuptools-20.10.1.zip?

jaraco · 2016-04-25T23:09:08Z

I will try to merge the PR and update the bootstrap bookmark.

dstufft · 2016-04-25T23:11:49Z

@AltxSF Sorry, can you do that one more time with a -v added to it as well?

dstufft · 2016-04-25T23:26:28Z

@AltxSF Try it now and see if it works.

dstufft · 2016-04-25T23:28:27Z

Ok good, misconfiguration in the CDN. Sorry about that!

ewang320 · 2016-04-26T02:12:23Z

@dstufft I might be missing something. What exactly do I need to bypass the error. I ran
curl https://bootstrap.pypa.io/ez_setup.py -o - | python and same error persists

ernix · 2016-04-26T04:01:58Z

@ewang320 try curl https://bootstrap.pypa.io/ez_setup.py -o - | python - --version=20.9.0

jaraco · 2016-04-26T04:11:49Z

I've merged the pull requests (#559 and #560) for the bootstrap script. Please report back if that doesn't correct the issue.

FrancescoRizzi · 2016-04-26T11:30:22Z

Sorry, I'm still getting the same error.
Downloaded ez_setup from here, but still get:

File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 807, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

jimfulton · 2016-04-26T12:20:11Z

Is that the full traceback?

Do you have curl or wget installed?

FrancescoRizzi · 2016-04-26T12:23:08Z

Sorry. Full output is here:

%> python ez_setup.py
Extracting in /var/folders/5f/zvqgl88j47zbqzcydcncpzz8nvzfxy/T/tmpQjnOme
Traceback (most recent call last):
  File "ez_setup.py", line 415, in <module>
    sys.exit(main())
  File "ez_setup.py", line 412, in main
    return _install(archive, _build_install_args(options))
  File "ez_setup.py", line 52, in _install
    with archive_context(archive_filename):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "ez_setup.py", line 103, in archive_context
    with ContextualZipFile(filename) as archive:
  File "ez_setup.py", line 87, in __new__
    return zipfile.ZipFile(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 766, in __init__
    self._RealGetContents()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 807, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

Also, I have used the suggested curl to get v.20.9 so I'm moving on in the meanwhile, but will try and get 20.10 asap. Thanks

jaraco · 2016-04-26T14:04:47Z

@FrancescoRizzi, I think the issue might be that you have a bad download locally. If ez_setup.py finds a local copy, it doesn't attempt to download it again. If you remove the local (probably small) zip file and re-run the script, I expect it will work. I've tested the latest bootstrap script on OS X and Ubuntu (with and without curl) and the download executes normally as expected.

jaraco · 2016-04-26T14:22:54Z

In #562, I also confirmed the fix works for Windows.

idgserpro · 2016-04-26T14:59:25Z

@jaraco When we use bootstrap-buildout.py, and there's a bad download locally, it doesn't matter because setuptools is always downloaded: if you want to use a local copy, you need to explicitly call

python bootstrap-buildout.py --setuptools-to-dir=.

Shouldn't we have the same behavior in ez_setup.py? Always download but give the option to use a local copy, if needed? We're asking this because this "bad download" error is really confusing for beginners, since this is the official primary method of installation.

Anyways, thanks for the fast fix.

jaraco · 2016-04-26T15:41:20Z

Shouldn't we have the same behavior in ez_setup.py?

@idgserpro: I agree. Would you ticket it and consider making a PR?

FrancescoRizzi · 2016-04-26T17:14:37Z

@jaraco thanks for the info. Where would ez_setup find a bad local copy "locally"?
I'm on a Mac OS X El Capitan (10.11.4), and ez_setup was running in an empty directory just created as virtualenv (specifically via virtualenvwrapper mkproject).

I tried to create a new virtualenv, and it seems to install setuptools-18.2.dist-info right away (at least lssitepackages shows that).
For added fun, if I'm out of any virtualenv and do a pip list, I find setuptools (20.3.1)

idgserpro · 2016-04-26T17:23:26Z

@FrancescoRizzi Alongside ez_setup.py itself. And in this line, if there's already a local file, it doesn't download it again.

We tested with vanilla python ez_setup.py, don't know exactly what could be different from a virtualenv like yourself.

@jaraco Done. Check #563. We we'll try to do a PR.

jimfulton · 2016-04-26T18:05:16Z

My suspicion is that @FrancescoRizzi 's ez_setup is deciding not to use wget or curl and is falling back to the "insecure" code and that that code isn't handling the redirect properly.

I'm going to play with this later when I have some time. It might though be helpful for @FrancescoRizzi to add a pdb.sey_trace() to ez_setup.py to walk through what it's actually doing.

idgserpro · 2016-04-26T19:39:35Z

@jaraco I didn't follow setuptools bitbucket-github migration, but this issue was created 5 days ago in bitbucket https://bitbucket.org/pypa/pypi/issues/438/backwards-compatible-un-hashed-package. Are you planning to keep both repositories? Since you migrated the issues to github as well, wouldn't it be the case to at least close bitbucket issues? Just my 0.02c.

jaraco · 2016-04-26T19:47:08Z

@idgserpro: That issue was created against the PyPI project, not Setuptools. A distinction which I missed initially as well. Setuptools issues are marked private, so aren't visible except to developers.

FrancescoRizzi · 2016-04-27T11:40:52Z

Hi.. I gave it another twirl this morning. Sadly I'm still getting the Bad Zip Error.

I tried it in 3 separate virtualenvs:

a virtualenv that existed and had setuptools 18.2, but from which I removed everything that seemed related to setuptools and/or easy_install (both from the project and virtualenv/lib/python2.7/site-packages
a virtualenv that existed (had setuptools 18.2), where I didn't do the steps I did in 1
a new virtualenv, just created

In all 3 cases I grabbed ez_setup.py this morning, and ran python with it.

The stack trace from test3 (below) seems to indicate download_setuptools elected to actually download a new copy of setuptools (20.10.1)

Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-20.10.1.zip
Extracting in /var/folders/5f/zvqgl88j47zbqzcydcncpzz8nvzfxy/T/tmpJiCjcG
Traceback (most recent call last):
  File "ez_setup.py", line 415, in <module>
    sys.exit(main())
  File "ez_setup.py", line 412, in main
    return _install(archive, _build_install_args(options))
  File "ez_setup.py", line 52, in _install
    with archive_context(archive_filename):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "ez_setup.py", line 103, in archive_context
    with ContextualZipFile(filename) as archive:
  File "ez_setup.py", line 87, in __new__
    return zipfile.ZipFile(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 766, in __init__
    self._RealGetContents()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 807, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

In fact, at the end of this run, there is a (newly created/downloaded) setuptools-20.10.1.zip next to ez_setup.py, sized at 313 bytes. I opened it via double-click (on Mac) to peek, and found just a setuptools-20.10.1.zip.cpgz inside.

Am willing to add print statements (or pdb) to ez_setup and rerun, if we're looking for anything in particular.

idgserpro · 2016-04-27T12:08:25Z

@FrancescoRizzi maybe your network is making cache:

https://bootstrap.pypa.io/ez_setup.py

FrancescoRizzi · 2016-04-27T12:27:36Z

@idgserpro you got it!

So... a hard reload/refresh of the ez_setup source got me a new copy (which i recognized as such thanks to the added Maintained at https://github.com/pypa/setuptools/tree/bootstrap. comment at the top). Running that ez_setup seems to finally get me an acceptable setuptools-20.10.1

If I may suggest, I'd add a version number at the top of ez_setup (it may not change often, but when it does it may help in recognizing a specific version of the file).

Thanks though! Now off to hunt every copy of the old ez_setup I have locally :D

idgserpro · 2016-04-27T12:59:50Z

If I may suggest, I'd add a version number at the top of ez_setup (it may not change often, but when it does it may help in recognizing a specific version of the file).

Good idea.

@jaraco buildout does this with bootstrap.py.

jaraco · 2016-04-28T03:00:48Z

Why not just use the date of the file, rather than add additional steps to the change process (and potential for mismatch)?

idgserpro · 2016-04-28T15:14:01Z

What do you mean by "date of the file"?

idgserpro · 2016-04-28T18:17:40Z

Is ez_setup.py synced every 15 minutes to https://bootstrap.pypa.io/ez_setup.py like @dstufft suggested in buildout/buildout#86 (comment)?

jaraco · 2016-04-29T14:02:16Z

What do you mean by "date of the file"?

$ curl -s -I https://bootstrap.pypa.io/ez_setup.py | grep Last-Modified
Last-Modified: Tue, 26 Apr 2016 16:30:27 GMT

Annoyingly, Github doesn't seem to provide a Last-Modified date, but does provide an ETag header:

$ curl -s -I https://raw.githubusercontent.com/pypa/setuptools/bootstrap/ez_setup.py | grep ETag
ETag: "e05e52bf1d9cdeed93c16f6456e2c532f343fa22"

Is ez_setup.py synced every 15 minutes to ...

Yes.

idgserpro mentioned this issue Apr 25, 2016

Bootstrap fails with setuptools v20.10.1 buildout/buildout#293

Closed

jimfulton mentioned this issue Apr 25, 2016

Fix setuptools url and curl wants --location to follow redirects #559

Merged

buryat mentioned this issue Apr 25, 2016

bootstrap.py script errors (setuptools issue) nasa/RHEAS#42

Closed

olivielpeau mentioned this issue Apr 25, 2016

[source] Pin setuptools version DataDog/dd-agent#2442

Merged

jaraco closed this as completed Apr 26, 2016

jaraco reopened this Apr 26, 2016

jaraco changed the title ~~Version 20.10.x gives a "zipfile.BadZipfile: File is not a zip file" when running "python bootstrap-buildout.py"~~ Bootstrap script unable to locate releases following 20.10.0 release Apr 26, 2016

jaraco mentioned this issue Apr 26, 2016

cannot bootstrap any more using ez_setup.py, 404 for setuptools #562

Closed

jaraco closed this as completed Apr 26, 2016

idgserpro mentioned this issue Apr 26, 2016

Meaningful message if corrupted setuptools #563

Closed

JeffreyMFarley mentioned this issue Apr 26, 2016

vagrant up is broken cfpb/ckan-installer#23

Closed

idgserpro mentioned this issue Apr 28, 2016

booststrap-buildout is not compatible with the new links from pypi buildout/buildout#292

Closed

se1exin mentioned this issue May 1, 2016

Fix setuptools install issue. Rename pip3.4 to pip3 to avoid install errors torchbox/vagrant-django-template#16

Closed

gasman pushed a commit to torchbox/vagrant-django-template that referenced this issue May 12, 2016

Fix setuptools install issue. See pypa/setuptools#557

a2a7d47

gasman pushed a commit to torchbox/vagrant-django-template that referenced this issue May 12, 2016

Fix setuptools install issue. See pypa/setuptools#557

9c9ad9f

Bootstrap script unable to locate releases following 20.10.0 release #557

Bootstrap script unable to locate releases following 20.10.0 release #557

Comments

idgserpro commented Apr 25, 2016 • edited

sbocconi commented Apr 25, 2016

FrancescoRizzi commented Apr 25, 2016

cyrilgdn commented Apr 25, 2016

idgserpro commented Apr 25, 2016

idgserpro commented Apr 25, 2016

jimfulton commented Apr 25, 2016

foursue commented Apr 25, 2016

idgserpro commented Apr 25, 2016 • edited

jimfulton commented Apr 25, 2016

dstufft commented Apr 25, 2016

jaraco commented Apr 25, 2016

dstufft commented Apr 25, 2016

dstufft commented Apr 25, 2016

dstufft commented Apr 25, 2016

ewang320 commented Apr 26, 2016

ernix commented Apr 26, 2016

jaraco commented Apr 26, 2016 • edited

FrancescoRizzi commented Apr 26, 2016

jimfulton commented Apr 26, 2016

FrancescoRizzi commented Apr 26, 2016

jaraco commented Apr 26, 2016

jaraco commented Apr 26, 2016

idgserpro commented Apr 26, 2016

jaraco commented Apr 26, 2016

FrancescoRizzi commented Apr 26, 2016

idgserpro commented Apr 26, 2016

jimfulton commented Apr 26, 2016

idgserpro commented Apr 26, 2016

jaraco commented Apr 26, 2016

FrancescoRizzi commented Apr 27, 2016 • edited

idgserpro commented Apr 27, 2016

FrancescoRizzi commented Apr 27, 2016

idgserpro commented Apr 27, 2016

jaraco commented Apr 28, 2016

idgserpro commented Apr 28, 2016 • edited

idgserpro commented Apr 28, 2016

jaraco commented Apr 29, 2016

idgserpro commented Apr 25, 2016 •

edited

idgserpro commented Apr 25, 2016 •

edited

jaraco commented Apr 26, 2016 •

edited

FrancescoRizzi commented Apr 27, 2016 •

edited

idgserpro commented Apr 28, 2016 •

edited