Support Exclude Package with custom unsafe packages

[hmc-cs-mdrissi]

#333 Closes this issue. Custom unsafe packages allows you to exclude packages. I named the option --unsafe-packages as all logic is customizing UNSAFE_PACKAGES (motivated by this comment), but if people would prefer I call it exclude that works. The one difference in the two names is whether original UNSAFE_PACKAGES remain or are gone.

As I'd ideally like setuptools/pip/etc to be in my requirements.txt file, but want to exclude other packages I prefer some way to replace the existing unsafe packages entirely. This is not a big deal to me though so if people would prefer semantic of it only adds packages to unsafe list I can do that. That is less flexible though.

I also added a flag to control whether recursive dependencies of unsafe only packages are treated as unsafe or not. The current logic for setuptools/pip is recursive dependencies only from unsafe packages are treated unsafe, but my use case for exclude is I only want to remove the exact packages I list.

Multiple occurrences of the flag are allowed. It can be used like,

pip-compile --unsafe-package foo --unsafe-package bar ...

I added two test cases to cover recursive flag being on/off.

@nvie @ssbarnea

Contributor checklist

• Provided the tests for the changes.
• Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

• Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[ssbarnea]

LGTM and I even tested it on a project where I wanted to use that feature and worked as expected.

[atugushev]

Generally, it looks good. I have a concern though:

[di]

I see this is in the 6.5.0 milestone, is this blocking a 6.5.0 release?

[ssbarnea]

I would really want to see this feature in ASAP as lack of it seriously limited tool use in several projects.

Requested by @ssbarnea

[atugushev]

It does not block the 6.5.0. Let's release it later.

[hmc-cs-mdrissi]

Is thee anything that's recommended I should do for this PR? Should I deal with dependencies of dependencies logic or should I leave it as out of scope? If there's another commit I can do to help merge this that would be helpful. It'd be nice to use this without fork.

[ssbarnea]

Any chance to resolve current conflicts? I would really want to see this feature in.

[hmc-cs-mdrissi]

I've resolved conflicts and I think tests should pass now with 2020 resolver too.

[ssbarnea]

@atugushev Please have a look at this and if ok-ish lets get it in before it needs other rebases.

[atugushev]

Otherwise LGTM 👍🏻

[atugushev]

@hmc-cs-mdrissi thanks for cleaning up reverse_dependencies(). Could you also consider this hmc-cs-mdrissi#1?

[ssbarnea]

@atugushev I would merge it as is and let him fix after. Because that the first contribution even CI is not running until one of us is approving it.

[atugushev]

I would merge it as is and let him fix after. Because that the first contribution even CI is not running until one of us is approving it.

@ssbarnea sure! Feel free to.

[ssbarnea]

@atugushev That pip cache step is hilarous almost two hours and still running https://github.com/jazzband/pip-tools/runs/7379969275?check_suite_focus=true