0.32.0 passes_validation hangs on armel and ppc64el (or perhaps randomly) #6490
Labels
A-dependency-resolution
Area: dependency resolution and the resolver
A-testing-cargo-itself
Area: cargo's tests
C-bug
Category: bug
Comments
Eh2406
added
A-dependency-resolution
|
"150 minutes of inactivity" does not sound good. We have an assertion that prevents each case from running for more than 90 sec, in addition to the assertion that makes sure that completion for the case takes less than 60 sec. We run 256 cases, so it is possible to pass and take longer than 150 min but we don't see that in our CI (witch has a 10 min inactivity timeout). The full test would fail at the first time one of those gets hit, we disable shrinking in CI... Is it possible that you don't have the CI environment variable set? (Or one of the environment variables proptest provides for disabling shrinking.) |
|
Yes, for sure we don't have that envvar set - it's not obvious which ones you are supposed to set, when you're packaging hundreds of packages where cargo is just one of them. If I understand right, you are theorising that the 150 minutes is due to the test failing for some other reason, then the testsuite spending 150 minutes+ trying to shrink a test case? And that setting (FWIW I was not able to reproduce it "by hand" on a ppc64el machine when running only |
|
Thank you for the work on packaging, it sounds like a monumental task. The Cargo Team will do our best to make the work of packaging easier eany way we can. I am sorry I was dismissive. Is there an environment variable that gets set by default we can check for (like how we check for CI)? That is exactly my theorising. Also seeing a "real failure" increases the chance of us being able to make a cross platform reproducible case. |
|
No problem, I didn't think you were being dismissive. :) We do set some Debian-specific envvars for every build but I am not sure it's great for cargo to be checking those either. Let me think for a bit about the best approach and get back to you... |
|
I've set Regarding the |
|
OK, the "real" failure (or at least one of them) has happened again, and it seems like this is actually a variant of #6491. I'm going to change 90 -> 360 and try it again, hopefully we won't see it any more. Are there any more of these "timeouts" in the cargo tests, that should have a arch-based multiplier associated with them? |
|
This also raises an interesting issue with |
|
I set So why do we have something as non-deterministic as wall time in a test!? Dependency resolution is So why use shrinking at all if the test is non-deterministic!? The difference between exponential complexity and polynomial complexity is often dramatic enough that the variation of wall time is not relevant. So a large amount of the case can be simplified before shrinking stops working. |
bors
added a commit
that referenced
this issue
Jan 24, 2019
Some CI setups are much slower then the equipment used by Cargo itself This adds a "CARGO_TEST_SLOW_CPU_MULTIPLIER" that increases all the time outs used in the tests, and disables Proptest shrinking on non tty cases. Closes: #6491 CC: #6490, @infinity0
With a timeout-based failure, sometimes it might work and sometimes it might succeed. Especially when shrinking, you are trying to find the minimal case that "causes" a timeout so eventually it will hit up against a bunch of cases that "just go over" the timeout. But then there are so many variations it can do, all of which will non-deterministically "fail" or "succeed" since it is at the boundary of the timeout. So the shrinking "strategy" has nothing to go on, it carries on executing its shrinking strategy based on false assumptions, and it's not surprising that it will never ever finish. Furthermore each "timeout" takes that amount of time to fail, so the total test time would be |
|
Indeed shrinking gets inefficient as it approaches the threshold. My experience in practice is that it finishes if I run it overnight. When I have my head on straight I set it up to log the failing inputs during the shrinking process, then if I am not patient enough for it to terminate I can use the best one it has found so far. (I have been thinking about how to set this up to happen more automatically.) You can set a total-timeout, and maybe we should have a default, but I don't know when the diminishing returns will happen on the next unknown-unknown. |
|
Thanks for the report and the discussion. I am going to close for now. Feel free to continue the conversation here or open a new issue at any time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
All other tests pass ("... ok"), compared to output with a successful run. Eventually the build gets killed "E: Build killed with signal TERM after 150 minutes of inactivity" so probably this test is the problem.
Quite possibly related to #6258.
The text was updated successfully, but these errors were encountered: