Disallow failures for tests we expect to pass

[uranusjr]

Since we’re skipping tests with fails_on_new_resolver in group 1 and 2, disallowing them to fail would help us catch regressions.

[pfmoore]

Thanks for doing this, I didn't know how to get Travis to just allow failures in the one case.

[uranusjr]

And it’s immediately catching regressions…

[pfmoore]

Sigh.

Should I revert #8298 so we can get this in, or leave this on hold until we get a fix for #8298 sorted out? I'm inclined to revert, as we've now confirmed that having this in place is useful 🙂

[uranusjr]

The weird thing is those three tests are all passing locally on my machine, otherwise I could have just fixed them here 😕 So yeah maybe it’d be best to revert and work on investigating the problems in another PR.

[pradyunsg]

Yea, let's revert. :)

[pfmoore]

Reverting in #8304

[pfmoore]

The weird thing is those three tests are all passing locally on my machine

And the way the tests capture the subprocess output, and then assert and use the output as the assertion message, which pytest then abbreviates, makes it so hard to work out what went on. That's a really bad pattern we use in our tests, and I'd love to fix it somehow. But maybe not right now 🙂

[pfmoore]

This should probably be rebased on master before merging, to confirm that the tests now pass...

[pradyunsg]

fix it somehow

leans in from edge of a wall

Pssst! Look over here! Is more stuff like #8303 what you want?

[uranusjr]

🦶 I hope this passes now the output thing is fixed.

[pradyunsg]

https://travis-ci.org/github/pypa/pip/jobs/690706800 failed.

[uranusjr]

The test has been failing ever since #7996 was merged. It needs to somehow be fixed first. (It’s quite weird though; I imagine the logic should be entirely in the finder, which is the same between the two resolvers.)

[uranusjr]

Found the issue; we broke the candidate ordering PackageFinder gave us. Fix proposed in #8319.

[pradyunsg]

Let's add the GROUP=1 job to "allow failures" here, merge this, and remove that job in #8319? That'll reduce the need to do yet-another-rebase of this PR.

Note from 2-minutes-later-me: well, we'd need to rebase either PR if we want to ensure that CI passes the entire time. FWIW, we could merge that and then this PR and things will be hunky-dory, so... this entire comment adds basically no value and you just wasted your time reading this.

[pfmoore]

#8319 is merged now. Does this need rebasing to merge? I'm getting so confused... 🙁

Is there any way I can rebase someone else's PR? I feel like I'm spending my time getting @uranusjr to rebase stuff that I could quite happily do myself if only I knew how...

[uranusjr]

With the situation with the resolvelib PR in mind, I think we should just merge all the things first and take a complete look at the regressions. These test config PRs are blocking too many things…

[pradyunsg]

@pfmoore

git remote add uranusjr https://github.com/uranusjr/pip.git
git fetch uranusjr

git checkout disallow-failure-in-new-resolver
git rebase master
git push uranusjr disallow-failure-in-new-resolver -f

[pradyunsg]

I think we should just merge this and then make a follow up fixing the finer issues. Travis CI is already flaky/not-required, so this wouldn't break anything.

[uranusjr]

I don’t mind if you don’t.

[pradyunsg]

merge this and then make a follow up fixing the finer issues. Travis CI is already flaky/not-required, so this wouldn't break anything.

@pfmoore If you're OK with this plan, let's do that. Feel free to click merge on this one. :P