Tarball URL erased when compiled

[AstraLuma]

# requirements.in
#dramatiq[rabbitmq,watch]~=1.11
https://github.com/AstraLuma/dramatiq/archive/refs/heads/injectables.tar.gz#egg=dramatiq[rabbitmq,watch]
flask-melodramatiq

becomes a few different wrong things.

(This is a modified fork because I've been waiting for the PR to be reviewed for several months.)

Environment Versions

1. OS Type: Mac
2. Python version: 3.9.7
3. pip version: None (installed with pipx, not using pip-sync)
4. pip-tools version: 6.4.0

Steps to replicate

1. Use the above requirements.in snippet
2. Compile

Expected result

Both extras and URL are preserved.

Actual result

One or the other has been dropped.

[AstraLuma]

Ok, I'm seeing inconsistent behavior.

In my full actual requirements file, it's dropping the URL.

But then copying it to a test file, I got it to drop the extras.

[AstraLuma]

So what I can easily replicate in tests is dropping the extras:

#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
#    pip-compile test.in
#
--index-url https://pypi-cache.corp/path/to/repo/

click==7.1.2
    # via flask
dramatiq @ https://github.com/AstraLuma/dramatiq/archive/refs/heads/injectables.tar.gz
    # via
    #   -r test.in
    #   flask-melodramatiq
flask==1.1.4
    # via flask-melodramatiq
flask-melodramatiq==0.3.9
    # via -r test.in
itsdangerous==1.1.0
    # via flask
jinja2==2.11.3
    # via flask
markupsafe==2.0.1
    # via jinja2
werkzeug==1.0.1
    # via flask

But what I'm struggling to replicate is the original behavior where it drops the URL and keeps the extras.

[richafrank]

Hi @AstraLuma ,

I think #1519 fixes this, but would you check too?

[AstraLuma]

@richafrank, no, it still drops the extras (like in the first comment).

It stopped dropping the URL when I reduced the dependencies (legacy project I'm doing some clean up on).

[AstraLuma]

I have some hints:

• The tarball had a lower version number than the pip package
• I removed one of dramatiq's dependencies from the top-level dependency list.

[richafrank]

Let me know if you disagree, but I think there are a two separate things going on here:

• The URL requirement for dramatiq is missing its extras in the output. This looks like a potentially intentional change in https://github.com/jazzband/pip-tools/pull/1453/files#r671595549.
• The dependencies introduced by the extras are missing. I still believe this is fixed by Instantiate a new accumulator InstallRequirement for combine_install_requirements output #1519 , which produces the following output:

#
# This file is autogenerated by pip-compile with python 3.8
# To update, run:
#
#    pip-compile
#
click==8.0.3
    # via flask
dramatiq @ https://github.com/AstraLuma/dramatiq/archive/refs/heads/injectables.tar.gz
    # via
    #   -r requirements.in
    #   flask-melodramatiq
flask==2.0.2
    # via flask-melodramatiq
flask-melodramatiq==0.3.9
    # via -r requirements.in
gevent==21.8.0
    # via watchdog-gevent
greenlet==1.1.2
    # via gevent
itsdangerous==2.0.1
    # via flask
jinja2==3.0.2
    # via flask
markupsafe==2.0.1
    # via jinja2
pika==1.2.0
    # via dramatiq
prometheus-client==0.12.0
    # via dramatiq
watchdog==2.1.6
    # via
    #   dramatiq
    #   watchdog-gevent
watchdog-gevent==0.1.1
    # via dramatiq
werkzeug==2.0.2
    # via flask
zope.event==4.5.0
    # via gevent
zope.interface==5.4.0
    # via gevent

# The following packages are considered to be unsafe in a requirements file:
# setuptools

Notice that pika et al are here, even though the extras are not mentioned in the dramatiq requirement. Only the output formatting is missing the extras.

Regarding the URL requirement's formatting, it looks like the grammar in https://www.python.org/dev/peps/pep-0508/ supports extras, if we want to add them. The test in https://github.com/jazzband/pip-tools/pull/1453/files#r671595549 fails with that change, so we should decide what behavior is preferred. @atugushev ?

#1520 changes the behavior and the test, if we want it.

[AstraLuma]

Yeah, those seem to be the reproducible issues.

I should probably start from the un-reduced requirements.in file and see where it stopped doing the "drop the url" thing.

[atugushev]

References with #1608 and #1613.