Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't work with extras_require and tests_require #107

Open
dvorapa opened this issue Oct 8, 2018 · 15 comments
Open

Doesn't work with extras_require and tests_require #107

dvorapa opened this issue Oct 8, 2018 · 15 comments

Comments

@dvorapa
Copy link

dvorapa commented Oct 8, 2018

I was a bit surprised, that pipdeptree can not handle either extra, or test dependencies. See for example:
Expected:

$ python -mpipdeptree
pipdeptree==0.13.0
  - pip [required: >=6.0.0, installed: 18.0]
pywikibot==3.0.dev0
  - crontab==0.22.2
  - flickrapi==2.4.0
    - requests [required: >=2.2.1, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-oauthlib [required: >=0.4.0, installed: 1.0.0]
      - oauthlib [required: >=0.6.2, installed: 2.1.0]
      - requests [required: >=2.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-toolbelt [required: >=0.3.1, installed: 0.8.0]
      - requests [required: >=2.0.1,<3.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: >=1.5.2, installed: 1.11.0]
  - google==2.0.1
    - beautifulsoup4 [required: Any, installed: 4.6.3]
  - irc==16.4
    - jaraco.collections [required: Any, installed: 1.5.3]
      - jaraco.classes [required: Any, installed: 1.5]
        - six [required: Any, installed: 1.11.0]
      - jaraco.text [required: Any, installed: 1.10.1]
        - jaraco.functools [required: Any, installed: 1.20]
          - more-itertools [required: Any, installed: 4.3.0]
            - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - six [required: >=1.7.0, installed: 1.11.0]
    - jaraco.functools [required: >=1.20, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - jaraco.itertools [required: >=1.8, installed: 2.5.2]
      - inflect [required: Any, installed: 1.0.1]
      - more-itertools [required: >=4.0.0, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - six [required: Any, installed: 1.11.0]
    - jaraco.logging [required: Any, installed: 1.5.2]
      - six [required: Any, installed: 1.11.0]
      - tempora [required: Any, installed: 1.13]
        - jaraco.functools [required: >=1.20, installed: 1.20]
          - more-itertools [required: Any, installed: 4.3.0]
            - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
        - pytz [required: Any, installed: 2018.5]
        - six [required: Any, installed: 1.11.0]
    - jaraco.stream [required: Any, installed: 1.2]
      - six [required: Any, installed: 1.11.0]
    - jaraco.text [required: Any, installed: 1.10.1]
      - jaraco.collections [required: Any, installed: 1.5.3]
        - jaraco.classes [required: Any, installed: 1.5]
          - six [required: Any, installed: 1.11.0]
        - six [required: >=1.7.0, installed: 1.11.0]
      - jaraco.functools [required: Any, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - more-itertools [required: Any, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.5]
    - six [required: Any, installed: 1.11.0]
    - tempora [required: >=1.6, installed: 1.13]
      - jaraco.functools [required: >=1.20, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - pytz [required: Any, installed: 2018.5]
      - six [required: Any, installed: 1.11.0]
  - memento-client==0.6.1
    - requests [required: >=2.7.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - mwoauth==0.3.2
    - oauthlib [required: Any, installed: 2.1.0]
    - PyJWT [required: >=1.0.1,<2.0.0, installed: 1.6.4]
    - requests [required: Any, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-oauthlib [required: Any, installed: 1.0.0]
      - oauthlib [required: >=0.6.2, installed: 2.1.0]
      - requests [required: >=2.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: Any, installed: 1.11.0]
  - mwparserfromhell==0.5.1
  - Pillow==5.3.0
  - pycountry==18.5.26
  - pydot==1.2.4
    - pyparsing [required: >=2.1.4, installed: 2.2.2]
  - PyMySQL==0.9.2
    - cryptography [required: Any, installed: 2.3.1]
      - asn1crypto [required: >=0.21.0, installed: 0.24.0]
      - cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
        - pycparser [required: Any, installed: 2.19]
      - idna [required: >=2.1, installed: 2.7]
      - six [required: >=1.4.1, installed: 1.11.0]
  - python-stdnum==1.9
  - requests [required: >=2.9,!=2.18.2, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - sseclient==0.0.19
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: Any, installed: 1.11.0]

Current:

$ python -mpipdeptree
crontab==0.22.2
flickrapi==2.4.0
  - requests [required: >=2.2.1, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-oauthlib [required: >=0.4.0, installed: 1.0.0]
    - oauthlib [required: >=0.6.2, installed: 2.1.0]
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-toolbelt [required: >=0.3.1, installed: 0.8.0]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: >=1.5.2, installed: 1.11.0]
google==2.0.1
  - beautifulsoup4 [required: Any, installed: 4.6.3]
irc==16.4
  - jaraco.collections [required: Any, installed: 1.5.3]
    - jaraco.classes [required: Any, installed: 1.5]
      - six [required: Any, installed: 1.11.0]
    - jaraco.text [required: Any, installed: 1.10.1]
      - jaraco.functools [required: Any, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - six [required: >=1.7.0, installed: 1.11.0]
  - jaraco.functools [required: >=1.20, installed: 1.20]
    - more-itertools [required: Any, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - jaraco.itertools [required: >=1.8, installed: 2.5.2]
    - inflect [required: Any, installed: 1.0.1]
    - more-itertools [required: >=4.0.0, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - six [required: Any, installed: 1.11.0]
  - jaraco.logging [required: Any, installed: 1.5.2]
    - six [required: Any, installed: 1.11.0]
    - tempora [required: Any, installed: 1.13]
      - jaraco.functools [required: >=1.20, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - pytz [required: Any, installed: 2018.5]
      - six [required: Any, installed: 1.11.0]
  - jaraco.stream [required: Any, installed: 1.2]
    - six [required: Any, installed: 1.11.0]
  - jaraco.text [required: Any, installed: 1.10.1]
    - jaraco.collections [required: Any, installed: 1.5.3]
      - jaraco.classes [required: Any, installed: 1.5]
        - six [required: Any, installed: 1.11.0]
      - six [required: >=1.7.0, installed: 1.11.0]
    - jaraco.functools [required: Any, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - more-itertools [required: Any, installed: 4.3.0]
    - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - pytz [required: Any, installed: 2018.5]
  - six [required: Any, installed: 1.11.0]
  - tempora [required: >=1.6, installed: 1.13]
    - jaraco.functools [required: >=1.20, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.5]
    - six [required: Any, installed: 1.11.0]
memento-client==0.6.1
  - requests [required: >=2.7.0, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
mwoauth==0.3.2
  - oauthlib [required: Any, installed: 2.1.0]
  - PyJWT [required: >=1.0.1,<2.0.0, installed: 1.6.4]
  - requests [required: Any, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-oauthlib [required: Any, installed: 1.0.0]
    - oauthlib [required: >=0.6.2, installed: 2.1.0]
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: Any, installed: 1.11.0]
mwparserfromhell==0.5.1
Pillow==5.3.0
pipdeptree==0.13.0
  - pip [required: >=6.0.0, installed: 18.0]
pycountry==18.5.26
pydot==1.2.4
  - pyparsing [required: >=2.1.4, installed: 2.2.2]
PyMySQL==0.9.2
  - cryptography [required: Any, installed: 2.3.1]
    - asn1crypto [required: >=0.21.0, installed: 0.24.0]
    - cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
      - pycparser [required: Any, installed: 2.19]
    - idna [required: >=2.1, installed: 2.7]
    - six [required: >=1.4.1, installed: 1.11.0]
python-stdnum==1.9
pywikibot==3.0.dev0
  - requests [required: >=2.9,!=2.18.2, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
sseclient==0.0.19
  - requests [required: >=2.0.0, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: Any, installed: 1.11.0]
@naiquevin
Copy link
Contributor

Hi, I didn't get what you exactly mean by "handle" extra/test dependencies. It would be helpful if you could please add just the line diff between expected and actual output.

@dvorapa
Copy link
Author

dvorapa commented Oct 9, 2018

Hi, I didn't get what you exactly mean by "handle" extra/test dependencies. It would be helpful if you could please add just the line diff between expected and actual output.

The example shows 20 packages. But 18 of them were installed using pip install -e pywikibot[extras] (or pip install -r requirements.txt) (as extra dependencies of pywikibot) and therefore should be marked as dependencies of pywikibot. They are not. They are marked as explicitly installed packages, which they are not. extras_require and test_require are parts of pip specification, both containing dependencies similar to install_require (which contains the dependencies pipdeptree can work with).

This issue is a huge limitation as it does not mark dependencies as dependencies and therefore the tree is not correct.

@naiquevin
Copy link
Contributor

Ok, got it now. Thanks for the clarification.

pipdeptree uses pip's internals to build the graph. Will have to check if pip exposes additional metadata for extras and test dependencies. Only in that case, it can be supported.

@naiquevin naiquevin added the tobeconfirmed To be confirmed after more investigation label Jul 14, 2019
@AWhetter
Copy link

The DistInfoDistribution.requires() (the class that pip returns from get_installed_distributions()) takes an extras argument to specify which extras to query for requirements.

from pip._internal.utils.misc import get_installed_distributions
help(get_installed_distributions()[13].requires)

Each Requirement object returned from requires() has an extras that says which extras the requirement comes from.

get_installed_distributions()[13].requires()[2].extras

It should be possible to use these to store extras information in the tree somehow.

For a smaller test case, installing jira gives the following deptree:

cryptography==2.7
  - asn1crypto [required: >=0.21.0, installed: 0.24.0]
  - cffi [required: >=1.8,!=1.11.3, installed: 1.12.3]
    - pycparser [required: Any, installed: 2.19]
  - enum34 [required: Any, installed: 1.1.6]
  - ipaddress [required: Any, installed: 1.0.22]
  - six [required: >=1.4.1, installed: 1.12.0]
jira==2.0.0
  - defusedxml [required: Any, installed: 0.6.0]
  - oauthlib [required: >=1.0.0, installed: 3.0.2]
  - pbr [required: >=3.0.0, installed: 5.4.1]
  - requests [required: >=2.10.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.6.16]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - requests-oauthlib [required: >=0.6.1, installed: 1.2.0]
    - oauthlib [required: >=3.0.0, installed: 3.0.2]
    - requests [required: >=2.0.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - requests-toolbelt [required: Any, installed: 0.9.1]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - setuptools [required: >=20.10.1, installed: 41.0.1]
  - six [required: >=1.10.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.2.1]
PyJWT==1.7.1
wheel==0.33.4

Jira actually depends on oauthlib[signedtoken] (ie oauthlib with the signedtoken extra), which depends on cryptography and PyJWT. Because pipdeptree does not detect either as a dependency of the oauthlib extra, they are incorrectly listed as top level packages.

@naiquevin
Copy link
Contributor

@AWhetter: This is helpful. Thanks!

@wimglenn
Copy link

@naiquevin Note that since Python 3.8 you can use importlib.metadata to get this "extra" info easily:

>>> from importlib.metadata import requires
>>> requires("jira")
['defusedxml', 'oauthlib[signedtoken] (>=1.0.0)', 'pbr (>=3.0.0)', 'requests-oauthlib (>=0.6.1)', 'requests (>=2.10.0)', 'requests-toolbelt', 'setuptools (>=20.10.1)', 'six (>=1.10.0)', "argparse; (python_version<'2.7')", "requests-futures (>=0.9.7); extra == 'async'", 'ipython (<6.0.0,>=4.0.0); python_version < "3.3" and extra == \'cli\'', 'ipython (>=4.0.0); python_version >= "3.3" and extra == \'cli\'', "filemagic (>=1.6); extra == 'opt'", "PyJWT; extra == 'opt'", "requests-jwt; extra == 'opt'", "requests-kerberos; extra == 'opt'"]
>>> requires("oauthlib")
["cryptography ; extra == 'rsa'", "blinker ; extra == 'signals'", "cryptography ; extra == 'signedtoken'", "pyjwt (>=1.0.0) ; extra == 'signedtoken'"]

@AWhetter pip._internal should not be imported. It's private API.

@flying-sheep
Copy link

Is there an alternative that supports this?

@flying-sheep
Copy link

flying-sheep commented Aug 17, 2021

We can use graphviz’ shape=record for this. One real life example (simplified):

digraph structs {
    node [shape=record]
    edge [color="#00000077"]
    
    api    [label="{ <base> api    | { <test> test | <dev> dev | <testing> testing }}"]
    server [label="{ <base> server | { <test> test | <dev> dev | <testing> testing }}"]
    utils  [label="{ <base> utils  | { <test> test | <dev> dev }}"]
    pytest [label="<base> pytest"]

    api:base -> utils:base
    server:base -> utils:base

    api:testing -> server:testing
    utils:test -> pytest:base
    api:test -> pytest:base
    server:test -> pytest:base
    server:testing -> pytest:base
}

grafik

Here, each packages’ tests depend on pytest, whereas both api and server have a extra called testing (containing test utils that other packages can use).

pipdeptree just ignores that api has an optional dependency on server, leaving out the actually important information I would have needed to debug a nasty dependency issue. I would have needed an output like this:

$ pipdeptree -p api[testing] --graph-output=dot | xdot /dev/stdin

grafik

@naiquevin
Copy link
Contributor

There's an open PR for supporting extra requirements - #138 , which basically builds on top of another PR #132. I haven't been able to spend much time on it after the initial work. IIRC, most changes are done but I don't want to haste the release of it as there's a chance of introducing regressions.

Let me rebase that branch with latest master over this weekend so that it's at least usable on experimental basis.

@dill0wn
Copy link

dill0wn commented Nov 8, 2021

I just ran into this issue today. My use case is actually in a python script, so i'll take a look at importlib.metadata as suggested in the earlier comment #107 (comment)

EDIT: Nevermind, I'm still on 3.7 👎 so I can't.

@flying-sheep
Copy link

You can use importlib_metadata, the backport on PyPI

@ralphie0112358
Copy link

Do you plan to fix this issue?

@naiquevin
Copy link
Contributor

naiquevin commented Mar 1, 2022 via email

@gaborbernat
Copy link
Member

gaborbernat commented Sep 4, 2022

A PR trying to address this would be welcome! tests_require is not supported but extra should be.

@Kagami
Copy link

Kagami commented Jul 2, 2023

There's similar issue with jupyterlab: one of its dependencies jupyter_events has jsonschema[format-nongpl] as dependency which depends on:

format-nongpl = [
  "fqdn",
  "idna",
  "isoduration",
  "jsonpointer>1.13",
  "rfc3339-validator",
  "rfc3986-validator>0.1.0",
  "uri_template",
  "webcolors>=1.11",
]

so you have a lot of extra top-level dependencies in the output:

$ pipdeptree -u -d 0
fqdn==1.5.1
isoduration==20.11.0
jsonpointer==2.4
jupyterlab==4.0.2
uri-template==1.3.0
webcolors==1.13

Note that only jupyterlab here is the only real top-level dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants