pip-audit not auditing the same package versions as pip installs.

[skilleter]

Bug description

My understanding of pip-audit is that, when given a requirements.txt file, it would audit the same versions of the same packages that pip would install. This doesn't seem to be the case.

I have a requirements.txt file containing just the cryptography module with no version constraint specified.

If I create a venv and install the package then it installs:

• cryptography 36.0.2
• cffi 1.15.0 (as cryptography has a requirement for cffi>=1.1.2)
• pcparser 2.21

If I run pip-audit -r requirement.txt then it tries to install cffi version 1.0.2-2 rather than 1.15.0 and fails with:

pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/tmp/tmpffo5omkp/bin/python3', '-m', 'pip', 'install', '/tmp/tmp4dbeewpp/cffi-1.0.2-2.tar.gz']

If I change the requirements.txt file to contain cryptography==36.0.2, the same thing happens.

Likewise, if I append cffi==1.15.0 to the requirements.txt file pip-audit still tries, and fails, to use cffi 1.0.2-2

Reproduction steps

Running in Ubuntu 22.04 Docker container with just python3, venv, git installed.
Running as a non-root user in the container I install pip-audit, either via pip, or directly from the git repo
Create requirements.txt containing just cryptography
Create a venv, install cryptography package from requirements.txt using pip
Run pip list to confirm package versions installed as listed above (specifically cffi 1.15.0)
Run pip-audit -r requirements.txt
pip-audit tries to install cffi 1.0.2.2 and fails, as above

Expected behavior

pip-audit should audit the same packages and the same versions of the packages as pip install installs
pip-audit should not fail

Screenshots and logs

Dockerfile used:

# Start with up-to-date Ubuntu

FROM ubuntu:22.04

# User to run with

ARG BUILD_USER=build
ARG BUILD_UID=1000
ARG BUILD_GID=1000
ARG HOME_DIR=/home/build

# Update & upgrade, install minimal Python setup

RUN apt update
RUN apt upgrade --yes
RUN apt install --yes apt-utils
RUN apt install --yes python3 python3-pip python3-venv git

# Add the user

RUN adduser --disabled-password --gecos '' ${BUILD_USER}
USER ${BUILD_UID}

WORKDIR ${HOME_DIR}

# Install pip-audit

#RUN pip install --upgrade --no-warn-script-location pip-audit
RUN python3 -m pip install git+https://github.com/trailofbits/pip-audit

Contents of requirements.txt file:

cryptography

Output from pip list in venv after installing cryptography package:

Package      Version
------------ -------
cffi         1.15.0
cryptography 36.0.2
pip          22.0.2
pycparser    2.21
setuptools   59.6.0

Output from pip-audit -v -r requirements.txt:

DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], project_path=None, format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15, paths=[], verbose=True, fix=False, require_hashes=False, index_url='https://pypi.org/simple', extra_index_urls=[], skip_editable=False)
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cryptography" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cryptography HTTP/1.1" 301 118
DEBUG:cachecontrol.controller:Updating cache with response from "https://pypi.org/simple/cryptography"
DEBUG:cachecontrol.controller:Caching permanent redirect
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cryptography/" in the cache
DEBUG:cachecontrol.controller:Current age based on date: 153
DEBUG:cachecontrol.controller:Freshness lifetime from max-age: 600
DEBUG:cachecontrol.controller:The response is "fresh", returning cached response
DEBUG:cachecontrol.controller:600 > 153
DEBUG:cachecontrol.controller:Looking up "https://files.pythonhosted.org/packages/5d/a9/b73a5d6f50a7b2f6ef65a2d2a14e848b62dfc79d10d29277586a94cf1f23/cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/5d/a9/b73a5d6f50a7b2f6ef65a2d2a14e848b62dfc79d10d29277586a94cf1f23/cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl HTTP/1.1" 200 4716191
DEBUG:cachecontrol.controller:Ignoring unknown cache-control directive: immutable
DEBUG:cachecontrol.controller:Updating cache with response from "https://files.pythonhosted.org/packages/5d/a9/b73a5d6f50a7b2f6ef65a2d2a14e848b62dfc79d10d29277586a94cf1f23/cryptography-36.0.2-cp36-abi3-macosx_10_10_universal2.whl"
DEBUG:cachecontrol.controller:etag object cached for 1209600 seconds
DEBUG:cachecontrol.controller:Caching due to etag
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cffi" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cffi HTTP/1.1" 301 110
DEBUG:cachecontrol.controller:Updating cache with response from "https://pypi.org/simple/cffi"
DEBUG:cachecontrol.controller:Caching permanent redirect
DEBUG:cachecontrol.controller:Looking up "https://pypi.org/simple/cffi/" in the cache
DEBUG:cachecontrol.controller:Current age based on date: 153
DEBUG:cachecontrol.controller:Freshness lifetime from max-age: 600
DEBUG:cachecontrol.controller:The response is "fresh", returning cached response
DEBUG:cachecontrol.controller:600 > 153
DEBUG:cachecontrol.controller:Looking up "https://files.pythonhosted.org/packages/ef/23/c6f7003ebb7b4b3fe4872f112b18ee181a3ec2b137e964093a8b35d4a5bd/cffi-1.0.2-2.tar.gz" in the cache
DEBUG:cachecontrol.controller:No cache entry available
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/ef/23/c6f7003ebb7b4b3fe4872f112b18ee181a3ec2b137e964093a8b35d4a5bd/cffi-1.0.2-2.tar.gz HTTP/1.1" 200 317417
DEBUG:cachecontrol.controller:Ignoring unknown cache-control directive: immutable
DEBUG:cachecontrol.controller:Updating cache with response from "https://files.pythonhosted.org/packages/ef/23/c6f7003ebb7b4b3fe4872f112b18ee181a3ec2b137e964093a8b35d4a5bd/cffi-1.0.2-2.tar.gz"
DEBUG:cachecontrol.controller:etag object cached for 1209600 seconds
DEBUG:cachecontrol.controller:Caching due to etag
Traceback (most recent call last):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_virtual_env.py", line 103, in post_setup
    run(package_install_cmd, state=self._state)
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_subprocess.py", line 51, in run
    raise CalledProcessError(f"{pretty_args} exited with {process.returncode}")
pip_audit._subprocess.CalledProcessError: python3 -m pip install /tmp/tmps9lu0wkz/cffi-1.0.2-2.tar.gz exited with 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/build/.local/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_cli.py", line 357, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_audit.py", line 66, in audit
    for dep, vulns in self._service.query_all(specs):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_service/interface.py", line 142, in query_all
    for spec in specs:
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/requirement.py", line 98, in collect
    for _, deps in self._resolver.resolve_all(iter(req_values)):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/interface.py", line 87, in resolve_all
    yield (req, self.resolve(req))
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 73, in resolve
    result = self.resolver.resolve([req])
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/home/build/.local/lib/python3.10/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 341, in get_dependencies
    return candidate.dependencies
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 119, in dependencies
    self._dependencies = list(self._get_dependencies())
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 101, in _get_dependencies
    deps: List[str] = self.metadata.get_all("Requires-Dist", [])
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 94, in metadata
    self._metadata = self._get_metadata_for_sdist()
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 162, in _get_metadata_for_sdist
    ve.create(ve_dir)
  File "/usr/lib/python3.10/venv/__init__.py", line 78, in create
    self.post_setup(context)
  File "/home/build/.local/lib/python3.10/site-packages/pip_audit/_virtual_env.py", line 105, in post_setup
    raise VirtualEnvError(f"Failed to install packages: {package_install_cmd}") from cpe
pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/tmp/tmpjk63y53b/bin/python3', '-m', 'pip', 'install', '/tmp/tmps9lu0wkz/cffi-1.0.2-2.tar.gz']

Platform information

• OS name and version: Ubuntu 22.04 (in Docker container)
• pip-audit version (pip-audit -V): pip-audit 2.1.0
• Python version (python -V or python3 -V): Python 3.10.3
• pip version (pip -V or pip3 -V): pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

[di]

Thanks for the report! Looks like there's a spurious sub-dependency here that we're not parsing correctly. When evaluating this for cffi:

https://github.com/trailofbits/pip-audit/blob/a9c12b9eb656103671528e9e9f007996623ea787/pip_audit/_dependency_source/resolvelib/pypi_provider.py#L305-L316

We get:

[
  <cffi-1-0-2==2 wheel=False>,
  <cffi==1.15.0 wheel=True>,
  <cffi==1.15.0 wheel=True>,
  <cffi==1.15.0 wheel=True>,
  <cffi==1.15.0 wheel=True>,
  <cffi==1.15.0 wheel=True>,
  ...
]

Where that first one is:

> candidates[0].name
'cffi-1-0-2'
> candidates[0].version
<Version('2')>

[woodruffw]

Yep, this looks specifically like a failure to handle "implicit post releases" as specified by PEP 440:

Implicit post releases
Post releases allow omitting the post signifier all together. When using this form the separator MUST be - and no other form is allowed. This allows versions such as 1.0-1 to be normalized to 1.0.post1. This particular normalization MUST NOT be used in conjunction with the implicit post release number rule. In other words, 1.0- is not a valid version and it does not normalize to 1.0.post0.

[woodruffw]

Did a bit of digging, and this looks like a bug in packaging.utils.parse_sdist_filename:

>>> from packaging.utils import parse_sdist_filename
>>> parse_sdist_filename("cffi-1.0.2-2.tar.gz")
('cffi-1-0-2', <Version('2')>)

[woodruffw]

Confirmed that it's the latest release of packaging (21.3), and that this is the buggy code:

def parse_sdist_filename(filename: str) -> Tuple[NormalizedName, Version]:
    if filename.endswith(".tar.gz"):
        file_stem = filename[: -len(".tar.gz")]
    elif filename.endswith(".zip"):
        file_stem = filename[: -len(".zip")]
    else:
        raise InvalidSdistFilename(
            f"Invalid sdist filename (extension must be '.tar.gz' or '.zip'):"
            f" {filename}"
        )

    # We are requiring a PEP 440 version, which cannot contain dashes,
    # so we split on the last dash.
    name_part, sep, version_part = file_stem.rpartition("-")
    if not sep:
        raise InvalidSdistFilename(f"Invalid sdist filename: {filename}")

    name = canonicalize_name(name_part)
    version = Version(version_part)
    return (name, version)

Specifically, the comment about "PEP 440 versions [not containing] dashes" is incorrect, since pre-normalized versions can. If I pass the version string directly into packaging.version.Version, it works correctly:

>>> from packaging.version import Version
>>> Version("1.0.2-2")
<Version('1.0.2.post2')>

[woodruffw]

Upstream: pypa/packaging#527

[di]

Given the age of the cffi distribution in question (circa 2015), perhaps it would be better for us to just assume that the parsing may not be guaranteed to be perfect and instead discard any candidates that don't match the distribution name we were expecting?

[woodruffw]

Yeah, that seems like a reasonable workaround. I can have a fix PR ready with that in a moment.

[woodruffw]

@skilleter I can confirm locally that #249 fixes this for me:

(env) work:pip-audit william$ pip-audit --format=json  -r ~/tmp/r3.txt
No known vulnerabilities found
{"dependencies": [{"name": "cryptography", "version": "36.0.2", "vulns": []}, {"name": "cffi", "version": "1.15.0", "vulns": []}, {"name": "pycparser", "version": "2.21", "vulns": []}], "fixes": []}

whre r3.txt:

cryptography

Can you also confirm? Once we do so, I'll push out a patch release 🙂

[woodruffw]

(Reopening for independent confirmation.)

[skilleter]

I can confirm that this now works for me, thanks very much.

[woodruffw]

No problem, thank you for confirming! I’ll cut a point release shortly. Sent from mobile. Please excuse my brevity.

…

On Mar 29, 2022, at 5:54 AM, John Skilleter ***@***.***> wrote:  I can confirm that this now works for me, thanks very much. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.

[woodruffw]

Cut with 2.1.1; should be available on PyPI shortly. Thanks again for reporting!