ansible-lint fails to install from source: AttributeError: 'SpecifierSet' object has no attribute 'split'

[ssbarnea]

Due to the complex workarounds collected in setup.py over the years ansible-lint may fail to install with something like:

pip install .                                                                                                                                                           [13:40:59]
Processing /Users/ssbarnea/.cache/pre-commit/repo5hdz7os2
ERROR: Command errored out with exit status 1:
command: /Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/setup.py'"'"'; __file__='"'"'/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/pip-egg-info
cwd: /private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/
Complete output (21 lines):
/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/git.py:68: UserWarning: "/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt" is shallow and may cause errors
warnings.warn('"%s" is shallow and may cause errors' % (wd.path,))
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/pip-req-build-zpowp0qt/setup.py", line 150, in <module>
__name__ == '__main__' and setuptools.setup(**setup_params)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/Users/ssbarnea/.pyenv/versions/3.7.4/lib/python3.7/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/dist.py", line 447, in __init__
k: v for k, v in attrs.items()
File "/Users/ssbarnea/.pyenv/versions/3.7.4/lib/python3.7/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/dist.py", line 735, in finalize_options
ep.load()(self, ep.name, value)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/dist.py", line 291, in check_specifier
packaging.specifiers.SpecifierSet(value)
File "/Users/ssbarnea/.cache/pre-commit/repo5hdz7os2/py3/lib/python3.7/site-packages/setuptools/_vendor/packaging/specifiers.py", line 594, in __init__
specifiers = [s.strip() for s in specifiers.split(",") if s.strip()]
AttributeError: 'SpecifierSet' object has no attribute 'split'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It happens that I already encountered this error with two other packages I maintain and the fix was to obliterate all workarounds from setup.py and go for safe setup.cfg approach.

References:

• https://github.com/pycontribs/selinux/blob/master/setup.py
• https://github.com/pycontribs/pytest-molecule/blob/master/setup.py
• SpecifierSet has no attribute split pypa/setuptools#1869 - same error another project
• https://bugs.launchpad.net/tripleo/+bug/1848512 - OpenStack Jobs broken by this bug

This is not fixed until:

• a new tag is added as pre-commit requires a tag (any tag)
• a release is made on pypi, so others can use it (even a new pre-release should do)

[webknjaz]

as pre-commit requires a tag

this is not true. you can use any "commitish", including SHA

[ssbarnea]

@webknjaz Almost forgot that. Yep could work. The downside is that pre-commit autoupdate will not work.

Btw, here is one of the failures we got on openstack projects: http://logs.rdoproject.org/58/23158/1/check/tox-linters/e49d829/job-output.txt

[shanedroid]

also encountered this with using ansible-lint as pre-commit hook in a github action job:

2019-10-17T15:45:50.7571288Z ##[group]Run pre-commit run -a
2019-10-17T15:45:50.7571436Z �[36;1mpre-commit run -a�[0m
2019-10-17T15:45:50.7613250Z shell: /bin/bash -e {0}
2019-10-17T15:45:50.7613418Z env:
2019-10-17T15:45:50.7613540Z   pythonLocation: /opt/hostedtoolcache/Python/3.7.4/x64
2019-10-17T15:45:50.7613664Z ##[endgroup]
2019-10-17T15:45:51.6659471Z [INFO] Initializing environment for https://github.com/IamTheFij/ansible-pre-commit.
2019-10-17T15:45:52.0082299Z [INFO] Initializing environment for https://github.com/ansible/ansible-lint.
2019-10-17T15:45:52.6020328Z [INFO] Initializing environment for https://github.com/prettier/prettier.
2019-10-17T15:45:54.1017508Z [INFO] Initializing environment for https://github.com/adrienverge/yamllint.
2019-10-17T15:45:54.4985598Z [INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks.
2019-10-17T15:45:54.9426053Z [INFO] Initializing environment for https://github.com/jumanjihouse/pre-commit-hooks:shellcheck.
2019-10-17T15:45:55.2525741Z [INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
2019-10-17T15:45:55.6409577Z [INFO] Installing environment for https://github.com/ansible/ansible-lint.
2019-10-17T15:45:55.6410798Z [INFO] Once installed this environment will be reused.
2019-10-17T15:45:55.6411247Z [INFO] This may take a few minutes...
2019-10-17T15:46:01.7001103Z An unexpected error has occurred: CalledProcessError: Command: ('/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/bin/python', '/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/bin/pip', 'install', '.')
2019-10-17T15:46:01.7002490Z Return code: 1
2019-10-17T15:46:01.7002927Z Expected return code: 0
2019-10-17T15:46:01.7003304Z Output: 
2019-10-17T15:46:01.7004022Z     Processing /home/runner/.cache/pre-commit/repoz8r2td9h
2019-10-17T15:46:01.7004515Z     
2019-10-17T15:46:01.7005248Z Errors: 
2019-10-17T15:46:01.7005676Z         ERROR: Command errored out with exit status 1:
2019-10-17T15:46:01.7007567Z          command: /home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-x17f_u4z/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-x17f_u4z/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-x17f_u4z/pip-egg-info
2019-10-17T15:46:01.7008597Z              cwd: /tmp/pip-req-build-x17f_u4z/
2019-10-17T15:46:01.7009075Z         Complete output (21 lines):
2019-10-17T15:46:01.7009940Z         /tmp/pip-req-build-x17f_u4z/.eggs/setuptools_scm-3.3.3-py3.7.egg/setuptools_scm/git.py:68: UserWarning: "/tmp/pip-req-build-x17f_u4z" is shallow and may cause errors
2019-10-17T15:46:01.7010756Z           warnings.warn('"%s" is shallow and may cause errors' % (wd.path,))
2019-10-17T15:46:01.7011249Z         Traceback (most recent call last):
2019-10-17T15:46:01.7032531Z           File "<string>", line 1, in <module>
2019-10-17T15:46:01.7058714Z           File "/tmp/pip-req-build-x17f_u4z/setup.py", line 150, in <module>
2019-10-17T15:46:01.7059229Z             __name__ == '__main__' and setuptools.setup(**setup_params)
2019-10-17T15:46:01.7059816Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
2019-10-17T15:46:01.7060042Z             return distutils.core.setup(**attrs)
2019-10-17T15:46:01.7060255Z           File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/distutils/core.py", line 108, in setup
2019-10-17T15:46:01.7060453Z             _setup_distribution = dist = klass(attrs)
2019-10-17T15:46:01.7061175Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/dist.py", line 447, in __init__
2019-10-17T15:46:01.7061663Z             k: v for k, v in attrs.items()
2019-10-17T15:46:01.7064777Z           File "/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/distutils/dist.py", line 292, in __init__
2019-10-17T15:46:01.7065031Z             self.finalize_options()
2019-10-17T15:46:01.7065732Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/dist.py", line 735, in finalize_options
2019-10-17T15:46:01.7065947Z             ep.load()(self, ep.name, value)
2019-10-17T15:46:01.7066662Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/dist.py", line 291, in check_specifier
2019-10-17T15:46:01.7067655Z             packaging.specifiers.SpecifierSet(value)
2019-10-17T15:46:01.7068297Z           File "/home/runner/.cache/pre-commit/repoz8r2td9h/py_env-python3.7/lib/python3.7/site-packages/setuptools/_vendor/packaging/specifiers.py", line 594, in __init__
2019-10-17T15:46:01.7068568Z             specifiers = [s.strip() for s in specifiers.split(",") if s.strip()]
2019-10-17T15:46:01.7069001Z         AttributeError: 'SpecifierSet' object has no attribute 'split'
2019-10-17T15:46:01.7069391Z         ----------------------------------------
2019-10-17T15:46:01.7069583Z     ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
2019-10-17T15:46:01.7069767Z     
2019-10-17T15:46:01.7069870Z 
2019-10-17T15:46:01.7070206Z Check the log at /home/runner/.cache/pre-commit/pre-commit.log
2019-10-17T15:46:01.7223163Z ##[error]Process completed with exit code 1.

😕

[ssbarnea]

@shanedroid Until we get the PRs reviewed, merged and release please use this workaround:

deps =
    # workaround for https://github.com/ansible/ansible-lint/issues/590
    virtualenv==16.3.0 # 16.7.6 not working
    pre-commit

Mainly this is forcing pre-commit to use a version of virtualenv that has an older setuptools, one that is still compatible with ansible-lint.

[cjeanner]

Heya,

Also hitting it, it's blocking tripleo-validations repository changes.....

2019-10-18 08:19:50.253059 | ubuntu-bionic | Errors:
2019-10-18 08:19:50.253189 | ubuntu-bionic |         ERROR: Command errored out with exit status 1:
2019-10-18 08:19:50.253940 | ubuntu-bionic |          command: /home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-q4STID/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-q4STID/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-q4STID/pip-egg-info
2019-10-18 08:19:50.254064 | ubuntu-bionic |              cwd: /tmp/pip-req-build-q4STID/
2019-10-18 08:19:50.254160 | ubuntu-bionic |         Complete output (19 lines):
2019-10-18 08:19:50.254268 | ubuntu-bionic |         Traceback (most recent call last):
2019-10-18 08:19:50.254368 | ubuntu-bionic |           File "<string>", line 1, in <module>
2019-10-18 08:19:50.254540 | ubuntu-bionic |           File "/tmp/pip-req-build-q4STID/setup.py", line 150, in <module>
2019-10-18 08:19:50.254692 | ubuntu-bionic |             __name__ == '__main__' and setuptools.setup(**setup_params)
2019-10-18 08:19:50.254923 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
2019-10-18 08:19:50.255022 | ubuntu-bionic |             return distutils.core.setup(**attrs)
2019-10-18 08:19:50.255143 | ubuntu-bionic |           File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
2019-10-18 08:19:50.255243 | ubuntu-bionic |             _setup_distribution = dist = klass(attrs)
2019-10-18 08:19:50.255461 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/dist.py", line 447, in __init__
2019-10-18 08:19:50.255543 | ubuntu-bionic |             k: v for k, v in attrs.items()
2019-10-18 08:19:50.255668 | ubuntu-bionic |           File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
2019-10-18 08:19:50.255746 | ubuntu-bionic |             self.finalize_options()
2019-10-18 08:19:50.256002 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/dist.py", line 735, in finalize_options
2019-10-18 08:19:50.256095 | ubuntu-bionic |             ep.load()(self, ep.name, value)
2019-10-18 08:19:50.256323 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/dist.py", line 291, in check_specifier
2019-10-18 08:19:50.256418 | ubuntu-bionic |             packaging.specifiers.SpecifierSet(value)
2019-10-18 08:19:50.256669 | ubuntu-bionic |           File "/home/zuul/.cache/pre-commit/repoSuKGCZ/py_env-python2/local/lib/python2.7/site-packages/setuptools/_vendor/packaging/specifiers.py", line 594, in __init__
2019-10-18 08:19:50.256800 | ubuntu-bionic |             specifiers = [s.strip() for s in specifiers.split(",") if s.strip()]
2019-10-18 08:19:50.256918 | ubuntu-bionic |         AttributeError: 'SpecifierSet' object has no attribute 'split'
2019-10-18 08:19:50.257007 | ubuntu-bionic |         ----------------------------------------
2019-10-18 08:19:50.257201 | ubuntu-bionic |     ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[LukeShortCloud]

According to the setuptools documentation, using setuptools.setup() will automatically load the setup.cfg file. The only gotcha is that it requires setuptools 30.3.0 which will not work Enterprise Linux 7 using the default python-setuptools RPM.

tl;dr? Using this for the ansible-lint setup.py allows me to install from git.

import setuptools

setuptools.setup()

Any reason why the setup.py is so complicated/hacky? For handling older versions of setuptools we could manually load the configuration using ini and then pass it to setuptools.setup().

[webknjaz]

@ekultails I'll tell you more: with the latest Pip and pyproject.toml you can even have setup.cfg w/o a setup.py existing in the repo at all.

But because of the list of supported envs we cannot rely on the latest versions of runtime pre-requisites.

That "hacks" in setup.py are only there to facilitate the backward compatibility.

Regarding "it won't work on RHEL": ideally, users should only install from wheels (or in case of RPMs, users should get pre-built artifacts anyway). This issue only hits cases where users build dist locally.

[ssbarnea]

But because of the list of supported envs we cannot rely on the latest versions of runtime pre-requisites.

AFAIK there is no such thing as a list of supported envs, or if exists is clearly not in this repository.

In the absence of an official list I would assume that CI/CD coverage provides that list.

Considering that in the last two weeks the CI of the project was broken because of this bug and that any of the two proposed patches is fixing it I would say that at this moment there is no tangible proof that this change breaks more than it fixes.

Ansible lint is also a pre-commit hook, and pre-commit is always installing from source. Mainly that is what is broken now. Any user using ansible-lint via pre-commit will encounter this bug.

As this seems to take more and more time to get fix, we may even end-up being forced to fork ansible-lint openstack, example https://review.opendev.org/#/c/689719/

[webknjaz]

1. Back when I was setting up packaging, I got some pushback regarding dropping envs so I'm not doing it w/o an explicit confirmation from Galaxy folks that it's fine. IMHO it's a tool for CIs and dev envs where the runtime is fully under control of the user so they may set up any required prerequisites that we'd ask them to. So yes, I understand why you want this and I'm not opposed to this idea as long as others are okay with this.
2. Just to reiterate, I think I mentioned it somewhere already, I'm sure that it's better to move the pre-commit hook definition into a separate repository and point it to PyPI so that it'd mostly hit wheels. (This would really solve a lot of problems)

[webknjaz]

there is no such thing as a list of supported envs

I was told that it's the same as Ansible Engine/Core

[ssbarnea]

While I do understand that the decision to drop the hacks needs to be measured well, I do not understand why the simple fix from #591 was not merged and was delayed.

The action here is to wait for the decision whether we can drop support for older envs.

How long until we get a decision? It would also be great to have some names of those that can make decisions. The reality is CI (azure) is broken on master and I did not see any PRs done from cores in attempt to fix it.

Regarding suggestion on #2 : pre-commit by design is installing all hooks using git clone, it cannot do it from wheels (at least not in native way). There is a way to define hooks that install wheels but that would break the ability to auto-update, not to mention that is likely to make execution unpredictable if user forgets to pin wheel version. Mainly I see #2 as a proposal to drop pre-commit support because current codebase no longer installs from source. I personally do not see it acceptable, especially as the issue is not caused by a bug in pre-commit or setuptools, it is solely coming from ansible-lint source code.

[asottile]

there is a way, it's just a bad idea

arguably a good thing happened here, we found out that the packaging of ansible-lint was broken pretty early

with shallow clones the amount of stuff saved isn't really impactful

[webknjaz]

@asottile the point is that I want to avoid pip building things where possible. If there was a repo with just a config but without setup.py, it'd be nice: I want that "default" flow that exists already to have pip use additional_dependencies instead of pip install .. So basically, I'm seeing for some dont_build_from_setup_py: true in the config, no other changes needed.

[asottile]

please read the thread above, this is addressed

[MartinThoma]

I just ran into the same error with

Python 3.7.5
>>> import setuptools; setuptools.__version__
'39.0.1'
>>> import virtualenv; virtualenv.__version__
'16.7.6'
>>> import pip; pip.__version__
'19.3.1'

Quick-Fix which worked for me:

pip install virtualenv==16.3.0 

[webknjaz]

@MartinThoma consider testing alpha pre-release v4.1.1a2 since we've dropped support for old setuptools and pip.

[bayandin]

Hi @webknjaz, is there an ETA of 4.1.1 release?
There is an open issue in 4.1.1 milestone, but I'm not sure that it is a real blocker: https://github.com/ansible/ansible-lint/issues?q=is%3Aopen+is%3Aissue+milestone%3A4.1.1

Thanks!

[webknjaz]

It's not my call to make. Ask @awcrosby.

[dnmvisser]

I just ran into the same error with

Python 3.7.5
>>> import setuptools; setuptools.__version__
'39.0.1'
>>> import virtualenv; virtualenv.__version__
'16.7.6'
>>> import pip; pip.__version__
'19.3.1'

Quick-Fix which worked for me:

pip install virtualenv==16.3.0 

just did some tests and 16.7.5 seems the last version to work with this.

[ssbarnea]

@dnmvisser To be honest, I am very aware of a bunch of broken virtualenv releases and unless you can reproduce the bug with the latest virtualenv version, I doubt anyone would step in to investigate.

[webknjaz]

I'm pretty sure he was installing some old version that had the setup.py shim.