-
-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Test Runner #3213
Comments
@sebastianbergmann great! I look forward to digesting (and working :) on these topics. |
@sebastianbergmann this looks like a major improvement, both for readability as well as performance. I'm wondering if the test collection shouldn't also be done as late as possible: That would prevent PHPUnit from loading all the value objects(those might be pretty big) beforehand. This would lead to a huge amount of yielding of course, but likely wouldn't make the code harder to read, you'd just end up iterating over a slightly different iterator in the end - one way smaller in memory most likely. |
@Idrinth That is definitely something I would like to explore once we have a "real" proof-of-concept. |
Probably off-topic, but still a big change... Would it be an idea to move all this to the
|
took the liberty of implementing a version of the discussed ideas as a tiny POC: https://github.com/Idrinth/phpunit-test-runner
The minimalistic repo should make it easier to have a quick look I hope. |
I'm wondering if dropping |
|
I do not use |
Great work |
Any news on this? |
I'm looking forward to seeing this in the future. Personally my biggest complaint about the current test runner comes from the inability to replace it with a custom one short of subclassing the Another problem, although maybe not directly caused by the runner itself, is the lack of a mechanism to register a PHPUnit Extension or Listener at runtime. For example this is useful when a listener depends on an object which itself needs constructor injection, which in some cases it's unfeasible using XML configuration alone. |
Supporting lazily loaded data providers is the most wanted feature for us :) |
Are there plans to support parallel test execution in the new runner implementation? |
@marcospassos working on that for v8, see #3736
@rask yes, parallel execution is something that is on my active to-do list. In v8 this could be achieved by extending the |
|
@sebastianbergmann a wholly good decision IMHO. Are there any "definite" plans or ideas in place to approach this in a more evolutionary manner? |
@rask I see @localheinz has already mentioned the new event subsystem. The current Myself I have quietly been cooking up a rebuild of the 'test execution reordering' functionality. It will move the business logic into the proper iterators inside the runner and remove the blackbox-like implementation. Benefits: the upcoming improvements for These large projects take a lot of time as the current implementation is a bit of a beast and the one-ping-only limit isn't helping either ;-) |
I see, sounds good. Will try and read up on the event subsystem. |
Introduction
This ticket supersedes #10.
Thanks to @epdenouden, the reordering of tests is a solved problem. This is something that I did not think possible without rewriting the test runner. And thanks to @Idrinth, it might be possible to delay the execution of data providers without rewriting the test runner.
However, as valuable as these contributions are, I do not think that fundamentally rewriting the test runner is something that can be avoided. The world of PHP has changed a lot in the 18 years since PHPUnit's inception: a lot of work(arounds) that is (are) done while finding and loading tests alone make(s) no more sense. Assumptions have changed (global variables are no longer important, for instance) and components such as PHP-Parser and BetterReflection are now available and remove the need to really load test classes before they are needed.
At its core, the current test runner has the same architecture and design as the one released as part of PHPUnit 2. Over the years, more and more features were added, layering workaround upon workaround to overcome the test runner's limitations. For almost two decades, this test runner has served us well (at least good enough). It's time to re-think test execution and come up with a new test runner that, hopefully, serves us just as well but without causing so many headaches when it comes to maintenance and the implementation of new features.
How tests are executed today
Before the first test is executed:
Test.php
(default)PHPUnit\Framework\TestCase
then each of itspublic
methods are introspected using the Reflection APItest
or if the method's docblock contains a@test
annotation then the method is considered a test method@dataProvider
annotation then the referenced data provider is executed and the next step is performed for each data set returned@dataProvider
is used)PHPUnit\Framework\TestSuite
objectThe above means that we have one object for each test that is to be executed before the first test is executed. These objects will remain in memory until the end of the PHP process that executed the test runner. Furthermore, data providers cannot leverage the benefits provided by generators when generators are used to implement data providers.
The above also means that data providers are executed and test objects are created even for tests that will not be executed later on because they are filtered, for instance using
--filter
,--group
, or--exclude-group
.The actual execution of tests is split across the
PHPUnit\Framework\TestCase::run()
,PHPUnit\Framework\TestResult::run()
,PHPUnit\Framework\TestCase::runBare()
, andPHPUnit\Framework\TestCase::runTest()
methods (see sequence diagram shown above). This is confusing and should be simplified.Another implementation aspect that should be simplified (by removing it) is
PHPUnit\Framework\TestSuite
. This object is a remnant from days long gone when PHPUnit was not able to search the filesystem for tests and required the manual addition of test classes toPHPUnit\Framework\TestSuite
objects in code.How tests could be executed in the future
Test.php
(default)PHPUnit\Runner\Test
value object is created for each test method foundPHPUnit\Runner\TestCollection
PHPUnit\Runner\TestCollection
to run the testsReordering and filtering tests, for instance, are operations that should be performed on the
TestCollection
after all tests have been collected and before the first test is executed.Here is an idea for what
PHPUnit\Runner\TestMethod
could look like:Here is an idea for how finding tests could be implemented:
A cache is used for avoiding the expensive static analysis for test sources that have not changed since the last execution of the test suite.
Executing a test could be as simple as this:
Of course, this initial prototype has no support for data providers, hook methods (
setUp()
,@before
, ...), etc. I am confident, though, that these can be implement with ease and in such a way that readability of the code does not diminish.The proof-of-concept code is available here.
The text was updated successfully, but these errors were encountered: