-
-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add functionality to test dependencies by changing test running order #3092
Conversation
…r, with and without dependency management.
… making it easy to reproduce results.
Codecov Report
@@ Coverage Diff @@
## test-reorder #3092 +/- ##
===============================================
Coverage 81.22% 81.22%
Complexity 2947 2947
===============================================
Files 110 110
Lines 7707 7707
===============================================
Hits 6260 6260
Misses 1447 1447
Continue to review full report at Codecov.
|
tests/Util/ConfigurationTest.php
Outdated
$this->assertFalse($_ENV['foo']); | ||
$this->assertEquals(1, \getenv('foo')); | ||
$this->assertEquals($fooValue, $_ENV['foo']); | ||
// $this->assertEquals(false, \getenv('foo')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this commented out intentionally?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catching that one. I did not yet know what to do with this one, $_ENV and getenv()/putenv() are not really equivalent. Setting $_ENV does not change the UNIX environment variables, but putenv() does.
Uncommenting this one makes the test fail. Do you have a preference?
src/Framework/TestCase.php
Outdated
/** | ||
* @param string $order | ||
*/ | ||
public function setTestRunningOrder($order): void |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use scalar type declarations when possible (like here). Also please do not use @param
annotations etc. when they provide no additional information.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
src/Framework/TestSuiteIterator.php
Outdated
|
||
/** | ||
* Reorder Tests within a TestCase in such a way as to resolve as many dependencies as possible. | ||
* The algorithm will leave the tests in original running order when it can. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about cross-class dependencies? They are possible, though I rarely see them "in the wild".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am aware of those, yes, although I have not encountered them. They are the reason I made this change: epdenouden@36697f0
Currently the business logic isn't smart enough yet to also sort the TestSuites based on development. The first work in the form of TestSuite::getDependencies()
is there.
Thank you, @epdenouden, for working on this. I will look into this more deeply as soon as I can. I hope we can get this into PHPUnit 7.2. |
@epdenouden I have moved the current state of this to a new branch: https://github.com/sebastianbergmann/phpunit/tree/test-reorder. Please rebase this pull request onto that branch and target that branch for future changes. Thanks! |
I am wondering whether it would make sense to move the reorder logic from Pseudo Code: <?php
class DependencyResolver extends ArrayIterator
{
public function __construct(TestSuiteIterator $tests)
{
$tests = iterator_to_array($tests);
// reorder $tests
parent::__construct($tests);
}
} |
@lstrojny This is probably interesting to you, too, considering "clever and smart". |
@sebastianbergmann Thanks for the quick replies! I will rebase and continue making the basic version clean and stable for 7.2. Good ideas for making this cleaner and more extensible. I am up for doing more coding for that. Re: the clever and fast logic: this project is actually the groundwork for something that we call the Breakfast extension, it automatically sorts the tests to run skipped, failed, etc first, but keeps their dependencies working. It also measures test and suite timings to make better sorting predictions. |
The "breakfast" thing sounds interesting. Lets get this sorted first, then have a look at that. Maybe it can also be integrated into the "core", maybe we can put it into an extension that uses the new, yet-to-be-added hook. |
Inspiration
Are your unit tests truly independent units?
In the past I have often been bitten by test suites that do not test what they claim to test. A common reason is hidden dependencies in the runtime context. Worst case this has caused applications to fail when deployed in production environments.
Changing the order of unit tests turns out to be a simple and effective way of bringing these assumptions out in the open.
Thanks to @Crazybus and @geogdog for providing real-world scenarios and feedback during development.
Summary of functionality
The test ordering functionality can be configured using command line parameters:
--reverse-order
run tests in reverse order--random-order
run tests in random order--random-order-seed=<number>
fix the random order, allowing random runs to be reproduced--ignore-dependencies
disable the bundled dependency resolverDesign decisions
Low impact, high performance
The feature will only wake up when there is work to do. The iterative dependency resolver is designed to do as little work as possible, using few iterations and minimal memory. It uses
@depends
annotations to keep as many dependent tests as possible in the required order.No unnecessary extra output is generated. Only the randomizer outputs the random seed using the default results printer.
Clean abstraction
The reordering is built into the main test runner and only changes the order of the tests of each test suite. Other parts of PHPUnit are not affected, with the exception of a bit of configuration handling.
Changes to the built-in testsuite
MultiDependencyTest
and updated existing tests to match--reverse-order
,--random-order
and--ignore-dependencies
@depends
for testGlobalsBackupPre() to PHPUnit's own testsuiteFuture work
phpunit.xml
in addition to the command line flags