puller.par not python 3 compatible

[Globegitter]

Now that container registry is python3 compatible the next problem seems to be in puller.par. Trying to build a docker image that needs to pull another image we get:

SyntaxError: invalid syntax
 (/usr/local/bin/python /path/to/external/puller/file/puller.par --directory /path/to/external/go_image_base/image --name gcr.io/distroless/base@sha256:bef8d030c7f36dfb73a8c76137616faeea73ac5a8495d535f27c911d0db77af3)and referenced by '//demo_app:demo-image'
(09:25:58) ERROR: Analysis of target '//demo_app:demo' failed; build aborted: no such package '@go_image_base//image': Pull command failed: Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/path/to/external/puller/file/puller.par/__main__.py", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/path/to/external/puller/file/puller.par/containerregistry/client/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/path/to/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 441, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_location
  File "/path/to/external/puller/file/puller.par/httplib2/__init__.py", line 942
    print "connect: (%s, %s) ************" % (self.host, self.port)
                                         ^
SyntaxError: invalid syntax
 (/path/to/external/puller/file/puller.par --directory /path/to/external/go_image_base/image --name gcr.io/distroless/base@sha256:bef8d030c7f36dfb73a8c76137616faeea73ac5a8495d535f27c911d0db77af3)
(09:25:58) INFO: Elapsed time: 2.811s
(09:25:58) INFO: 0 processes.

So the issue seems to be in puller.par's httplib2: httplib2/httplib2#29 I wonder if there is a solution around this, or if we really are blocked by that PR/Issue.

[Globegitter]

One option could be to switch over to https://github.com/google/go-containerregistry/blob/master/cmd/crane/doc/crane.md but I guess that could be quite an involved process to do in all places.

[siddharthab]

I am actually getting an error in the pusher too, but through containerregistry:

  File "/tmp/Bazel.runfiles_vdFAV3/runfiles/containerregistry/tools/fast_pusher_.py", line 29, in <module>
    from containerregistry.client import docker_creds
  File "/tmp/Bazel.runfiles_vdFAV3/runfiles/containerregistry/client/__init__.py", line 23, in <module>
    from containerregistry.client import docker_creds_
  File "/tmp/Bazel.runfiles_vdFAV3/runfiles/containerregistry/client/docker_creds_.py", line 31, in <module>
    import httplib2
  File "/tmp/Bazel.runfiles_vdFAV3/runfiles/httplib2/__init__.py", line 988
    raise socket.error, msg

I don't know how people have been able to push images with python3 with 0.0.28 release of containerregistry.

[siddharthab]

I wrote #481 for using pusher from go-containerregistry.

[nlopezgi]

Please see the PR that I sent (#490) that should provide a viable solution to avoid to have to make large changes to support this use case.

[Globegitter]

There is now also google/containerregistry#109 which should fix the issue.

[nlopezgi]

can we close this now? I think the path forward now is to migrate to go-containerregistry (#580). Please let me know if you want me to reopen for some reason.

[dgrnbrg]

Can you please help explain why this issue is closed? Is there a current solution to use python3 and rules_docker?

[nlopezgi]

Hi @dgrnbrg it closed because the plan (as indicated in the comment above) is to migrate to go-containerregistry which will solve this problem. Any reason why you want to keep it open?

[dgrnbrg]

I did not understand until your comment. Thank you for clarifying!

[mpictor]

FWIW if you have multiple versions of python installed, your distro may allow you to select between them with an env var. For example, when I hit this on Gentoo I found I could do EPYTHON=python2.7 bazel build //... to avoid the error with print in httplib2.

Assuming you're determined to not uninstall py3 or fiddle with defaults, another solution would be to add a dir with python symlink pointing to python2 as the first entry in PATH, so it's found before python3.

[Kernald]

Bazel also has a way to select a different Python binary than $(which python), by using the env variable BAZEL_PYTHON. Setting it to the path of a python2 executable should work, regardless of the distribution you're using. (#580 would be a way better fix, as it would just work out of the box, but that might help someone in the meantime.)