pip install docker-compose==1.25 fails due to lack of python headers for building subprocess32

[kostrzewa9ld]

Description of the issue

When running pip install docker-compose==1.25 either in virtual env or in centos:7 docker container pip tries to compile subprocess32 and requires python headers.

Context information (for bug reports)

Output of docker-compose version

1.25

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

(paste here)

Steps to reproduce the issue

1. virtualenv /tmp/foo
2. source /tmp/foo/bin/activate
3. pip install docker-compose==1.25

Observed result

Installation fails due to lack of python dev files.

Expected result

Installation succeeds and python dev files are not required.

Stacktrace / full error message

Running setup.py install for subprocess32 ... error
    ERROR: Command errored out with exit status 1:
     command: /tmp/foo/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"'; __file__='"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-YMsrMi/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/foo/include/site/python2.7/subprocess32
         cwd: /tmp/pip-install-OtSipE/subprocess32/
    Complete output (17 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying subprocess32.py -> build/lib.linux-x86_64-2.7
    running build_ext
    running build_configure
     _posixsubprocess_config.h is already up to date.
    building '_posixsubprocess32' extension
    creating build/temp.linux-x86_64-2.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-PPrPZj/python2.7-2.7.15=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
    _posixsubprocess.c:16:10: fatal error: Python.h: No such file or directory
     #include "Python.h"
              ^~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /tmp/foo/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"'; __file__='"'"'/tmp/pip-install-OtSipE/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-YMsrMi/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/foo/include/site/python2.7/subprocess32 Check the logs for full command output.

Additional information

OS version / distribution, docker-compose install method, etc.
CentOS 7 docker, Ubuntu 18.04.3 LTS

[plemelin]

I face the issue mentionned by @kostrzewa9ld and installed python-devel with yum. This allowed the installation of docker-compose to proceed. However, docker-compose will now fail with:

Traceback (most recent call last):
  File \"/usr/bin/docker-compose\", line 7, in <module>
    from compose.cli.main import main
  File \"/usr/lib/python2.7/site-packages/compose/cli/main.py\", line 23, in <module>
    from ..bundle import get_image_digests
  File \"/usr/lib/python2.7/site-packages/compose/bundle.py\", line 12, in <module>
    from .config.serialize import denormalize_config
  File \"/usr/lib/python2.7/site-packages/compose/config/__init__.py\", line 6, in <module>
    from .config import ConfigurationError
  File \"/usr/lib/python2.7/site-packages/compose/config/config.py\", line 50, in <module>
    from .validation import match_named_volumes
  File \"/usr/lib/python2.7/site-packages/compose/config/validation.py\", line 12, in <module>
    from jsonschema import Draft4Validator
  File \"/usr/lib/python2.7/site-packages/jsonschema/__init__.py\", line 33, in <module>
    import importlib_metadata as metadata
  File \"/usr/lib/python2.7/site-packages/importlib_metadata/__init__.py\", line 9, in <module>
    import zipp
  File \"/usr/lib/python2.7/site-packages/zipp.py\", line 12, in <module>
    import more_itertools
  File \"/usr/lib/python2.7/site-packages/more_itertools/__init__.py\", line 1, in <module>
    from more_itertools.more import *  # noqa
  File \"/usr/lib/python2.7/site-packages/more_itertools/more.py\", line 340
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax"

[ulyssessouza]

@plemelin Could you please provide more info? That would be great to open another bug report answering the template questions.

[plemelin]

@ulyssessouza

As I was writing how to reproduce the issue to you, it gave me the opportunity to rubber duck... The issue I'm facing (and many people will be facing at this point) is that the default version of pip installed by python-pip is version 8 (current version is 19...). Here is the bug report that woke me up: more-itertools/more-itertools#272

Here how to reproduce it easily (running this from my arch linux + docker machine):

sudo docker pull centos:centos7
sudo docker run --rm -it centos:centos7 /bin/bash

yum update -y
yum install -y epel-release
yum install -y python-pip

pip install docker-compose
# BAM. Fails with needing gcc

yum install -y gcc
pip install docker-compose
# BAM. Fails with needing Python.h

yum install -y python-devel
pip install docker-compose

docker-compose --version
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 7, in <module>
    from compose.cli.main import main
  File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 23, in <module>
    from ..bundle import get_image_digests
  File "/usr/lib/python2.7/site-packages/compose/bundle.py", line 12, in <module>
    from .config.serialize import denormalize_config
  File "/usr/lib/python2.7/site-packages/compose/config/__init__.py", line 6, in <module>
    from .config import ConfigurationError
  File "/usr/lib/python2.7/site-packages/compose/config/config.py", line 50, in <module>
    from .validation import match_named_volumes
  File "/usr/lib/python2.7/site-packages/compose/config/validation.py", line 12, in <module>
    from jsonschema import Draft4Validator
  File "/usr/lib/python2.7/site-packages/jsonschema/__init__.py", line 33, in <module>
    import importlib_metadata as metadata
  File "/usr/lib/python2.7/site-packages/importlib_metadata/__init__.py", line 9, in <module>
    import zipp
  File "/usr/lib/python2.7/site-packages/zipp.py", line 12, in <module>
    import more_itertools
  File "/usr/lib/python2.7/site-packages/more_itertools/__init__.py", line 1, in <module>
    from more_itertools.more import *  # noqa
  File "/usr/lib/python2.7/site-packages/more_itertools/more.py", line 340
    def _collate(*iterables, key=lambda a: a, reverse=False):
                               ^
SyntaxError: invalid syntax

Now, this lead me here: more-itertools/more-itertools#272
Which then let me fix the installation like so:

sudo docker pull centos:centos7
sudo docker run --rm -it centos:centos7 /bin/bash

yum update -y
yum install -y epel-release
yum install -y python-pip python-devel gcc
pip install --upgrade pip

pip install docker-compose
docker-compose --version
docker-compose version 1.25.0, build b42d419

[ulyssessouza]

@kostrzewa9ld Note that python-dev is mentioned in https://docs.docker.com/compose/install/ as dependency when installing via pip. Also note that it's an alternative installation.

[kostrzewa9ld]

@ulyssessouza
Just for the record as I guess it will be resolved by update of documentation, but as far as I can tell python-dev is only mentioned as a required dependency for alpine

For alpine, the following dependency packages are needed: py-pip, python-dev, libffi-dev, openssl-dev, gcc, libc-dev, and make.

What's more it only started to be required for non-alpine distros since 1.25, which is not even mentioned in this document:

The instructions below outline installation of the current stable release (v1.24.1) of Compose.

[jamshid]

I guess docker doesn't care because they're dropping python 2 support?
#6890

Or is Compose going away with Swarm?

FWIW pip3 seems to work fine:

yum install -y python3-pip
pip3 install docker-compose 

[nandogameiro]

This command solved the issue for me
sudo pip install 'more-itertools>=4.0.0,<=5.0.0' --force-reinstall

On this comment below user says more-itertools version >6.0 dropped support to python 2, so that's why need to downgrade to version 5.0.0

more-itertools/more-itertools#272 (comment)

[codesupremo]

I was able to resolved the subprocess32 issue after I installed gcc "yum install gcc" in CentOS 7

[ecorbett135]

Ran into this as well - this fixed it for me:

sudo yum install python-devel

[zouchengli]

@kostrzewa9ld Note that python-dev is mentioned in https://docs.docker.com/compose/install/ as dependency when installing via pip. Also note that it's an alternative installation

yum install -y epel-release
yum install -y python-pip python-devel gcc

OK

[zouchengli]

ERROR: Command errored out with exit status 1:
 command: /usr/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9mlRrh/subprocess32/setup.py'"'"'; __file__='"'"'/tmp/pip-install-9mlRrh/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-Z6XT_d/install-record.txt --single-version-externally-managed --compile --install-headers /usr/include/python2.7/subprocess32
     cwd: /tmp/pip-install-9mlRrh/subprocess32/
Complete output (62 lines):
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying subprocess32.py -> build/lib.linux-x86_64-2.7
running build_ext
running build_configure
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for unistd.h... (cached) yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking signal.h usability... yes
checking signal.h presence... yes
checking for signal.h... yes
checking sys/cdefs.h usability... yes
checking sys/cdefs.h presence... yes
checking for sys/cdefs.h... yes
checking for sys/types.h... (cached) yes
checking for sys/stat.h... (cached) yes
checking sys/syscall.h usability... yes
checking sys/syscall.h presence... yes
checking for sys/syscall.h... yes
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking for pipe2... yes
checking for setsid... yes
checking whether dirfd is declared... yes
configure: creating ./config.status
config.status: creating _posixsubprocess_config.h
building '_posixsubprocess32' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
_posixsubprocess.c:16:20: fatal error: Python.h: No such file or directory
 #include "Python.h"
                    ^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------

ERROR: Command errored out with exit status 1: /usr/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9mlRrh/subprocess32/setup.py'"'"'; file='"'"'/tmp/pip-install-9mlRrh/subprocess32/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-Z6XT_d/install-record.txt --single-version-externally-managed --compile --install-headers /usr/include/python2.7/subprocess32 Check the logs for full command output.

Because not install python-devel
yum -y install python-devel
Alter install docker-compose successfully
Successfully installed docker-compose-1.25.4 subprocess32-3.5.4 texttable-1.6.2

[zouchengli]

Running setup.py install for subprocess32 ... error
Complete output from command /usr/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-build-J19N4v/subprocess32/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-mbZyUM-record/install-record.txt --single-version-externally-managed --compile:
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying subprocess32.py -> build/lib.linux-x86_64-2.7
running build_ext
running build_configure
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in /tmp/pip-build-J19N4v/subprocess32': configure: error: no acceptable C compiler found in $PATH See config.log' for more details
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-J19N4v/subprocess32/setup.py", line 120, in
main()
File "/tmp/pip-build-J19N4v/subprocess32/setup.py", line 114, in main
'Programming Language :: Python :: Implementation :: CPython',
File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 53, in run
return _install.run(self)
File "/usr/lib64/python2.7/distutils/command/install.py", line 563, in run
self.run_command('build')
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
self.run_command(cmd_name)
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-build-J19N4v/subprocess32/setup.py", line 41, in run
self.run_command(command)
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-build-J19N4v/subprocess32/setup.py", line 26, in run
raise RuntimeError(configure_command + ' failed.')
RuntimeError: sh ./configure failed.

----------------------------------------

Command "/usr/bin/python2 -u -c "import setuptools, tokenize;file='/tmp/pip-build-J19N4v/subprocess32/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-mbZyUM-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-J19N4v/subprocess32/

Because not install gcc
yum -y install gcc
Alter install docker-compose successfully
Successfully installed PyYAML-5.3.1 attrs-19.3.0 backports.shutil-get-terminal-size-1.0.0 bcrypt-3.1.7 cached-property-1.5.1 cffi-1.14.0 contextlib2-0.6.0.post1 cryptography-2.9 docker-4.2.0 docker-compose-1.25.4 enum34-1.1.10 functools32-3.2.3.post2 importlib-metadata-1.6.0 jsonschema-3.2.0 paramiko-2.7.1 pathlib2-2.3.5 pycparser-2.20 pynacl-1.3.0 pyrsistent-0.16.0 scandir-1.10.0 subprocess32-3.5.4 unknown-0.0.0 unknown-0.0.0 websocket-client-0.57.0

[daleyzou]

Ran into this as well - this fixed it for me:

sudo yum install python-devel

Thanks, it solved my problem