Breakfast: rerun defects first #3147
Conversation
Codecov Report
@@ Coverage Diff @@
## master #3147 +/- ##
===========================================
+ Coverage 81.5% 82.1% +0.59%
- Complexity 3412 3506 +94
===========================================
Files 137 140 +3
Lines 9014 9218 +204
===========================================
+ Hits 7347 7568 +221
+ Misses 1667 1650 -17
Continue to review full report at Codecov.
|
|
sebastianbergmann
added
the
type/enhancement
|
Thanks for the quick review, again!
|
src/Util/TestResultCache.php
Outdated
| /** | ||
| * @var string | ||
| */ | ||
| public const DEFAULT_RESULT_CACHE_FILENAME = 'result_cache.json'; |
There was a problem hiding this comment.
if I have my project, eg ~/php-cs-fixer folder, and I would run phpunit with cache enabled, it would create ~/php-cs-fixer/result_cache.json ? if so, please let us make it phpunit.cache.json or sth like this
There was a problem hiding this comment.
I have no opinion on the filename of the default value. As a developer I want the following:
- Simple usage that just works in a basic code, test, repeat cycle. A file in the working directory works well.
- PHPUnit is part of automation: use the environment variable to do your devops and have it store the cache(s) for users/configs/build-unique-ID/git-hashes/etc however en whereever you want.
The default implementation of TestResultCache is there so the simplest use cases work out-of-the-box. The current cache is just a simple MVP. I would prefer NOT to use an external dependency like SQLite like PHPUnit Clever and Smart did, but I am starting to see why they went that way.
There was a problem hiding this comment.
well, here I'm even not talking about using SQLite or anything like that. (while Indeed, i can see value of it - yet in caching of PHP CS Fixer we don't use it as well - too fancy for our simple purpose).
what I'm saying here @epdenouden is that if I will run phpunit in my project, using cache and out-of-the-box approach (so I don't configure anything extra), phpunit will leave result_cache.json file there. if phpunit will leave sth, it shall indicate it's his leftover, putting it to the name part.
like
-result_cache
+phpunit_cacheThere was a problem hiding this comment.
@keradus quick reply: that was a good suggestion, thx! Have pushed a change for that.
I mixed two related topics regarding the caching, making my comment confusion. Adding in persistence cleanly has been more of a chore than I expected. Will spend more time tonight on a cleaner approach.
There was a problem hiding this comment.
new name looks shiny, thanks for the change!
then, I will try to review changes tomorrow ;)
src/Util/TestResultCache.php
Outdated
| return; | ||
| } | ||
|
|
||
| $this->defects = $cacheData['defects']; |
There was a problem hiding this comment.
we have no clue that decoded data contain such key...
and even if - what is the content of it.
it's optimistic approach here.
cache file could be treated as-is, one is not supposed to modify it manually.
maybe serialize and then unserialize with unserialize(..., ['allowed_classes' => CacheData::class]) ?
There was a problem hiding this comment.
Yes, the earliest versions of Breakfast were just that: patches to the code that quickly (un)serialized a results file to store the state between runs. I'll try some other options.
src/Util/TestResultCache.php
Outdated
|
|
||
| private function formatJSONOutputStruct(): string | ||
| { | ||
| return \json_encode([ |
There was a problem hiding this comment.
json_encode can fail as well (eg on non utf-8 input)
src/Util/TestResultCache.php
Outdated
| /** | ||
| * @var array | ||
| */ | ||
| private $times = []; |
There was a problem hiding this comment.
what is the content of those collections? array of.... ints? is key meaningful or numeric?
how single defect looks like ?
There was a problem hiding this comment.
Thanks, all good points regarding the results cache. Started refactoring that part to make it more robust and cleaner.
phpunit.xsd
Outdated
| @@ -227,6 +233,7 @@ | |||
| <xs:attribute name="printerClass" type="xs:string" default="PHPUnit\TextUI\ResultPrinter"/> | |||
| <xs:attribute name="printerFile" type="xs:anyURI"/> | |||
| <xs:attribute name="processIsolation" type="xs:boolean" default="false"/> | |||
| <xs:attribute name="stopOnDefect" type="xs:boolean" default="false"/> | |||
There was a problem hiding this comment.
would be nice to document somewhere what is a defect
There was a problem hiding this comment.
Thank you. I have added a description to the help message.
|
|
||
| private function updateRunState(\PHPUnit\Framework\Test $test, string $state): void | ||
| { | ||
| $testName = $this->fullTestName($test); |
src/Runner/TestSuiteSorter.php
Outdated
| ]; | ||
|
|
||
| /** | ||
| * @var array |
There was a problem hiding this comment.
Added description
src/Runner/TestSuiteSorter.php
Outdated
| /** | ||
| * @var array | ||
| */ | ||
| private $defectSortWeight = [ |
There was a problem hiding this comment.
Fixed, was a precalculated array before.
src/Runner/TestSuiteSorter.php
Outdated
| } | ||
| } | ||
|
|
||
| private function sort(TestSuite $suite, int $order, bool $resolveDependencies): void | ||
| public function setCache(TestResultCache $cache): void |
There was a problem hiding this comment.
I believe this property shall not be exposed
There was a problem hiding this comment.
You're right, the method is only used in the test. Will clean it up :)
| use PHPUnit\Framework\TestCase; | ||
| use PHPUnit\Framework\TestSuite; | ||
|
|
||
| class TestSuiteSorterEmptyTestCaseTest extends TestCase |
There was a problem hiding this comment.
there is no TestSuiteSorterEmptyTestCase class, how we can have test for it ?
test here shall be part of TestSuiteSorterTest test class, it's one concrete, edge scenario to be tested, not completely other class
There was a problem hiding this comment.
Refactored the fixtures in TestSuiteSorterTest and added this test in. Cleaner, thanks for the comment!
It started off as a quick test where I didn't want the fixtures from TestSuiteSorterTest. The name is indeed confusing, it was supposed to mean "TestSuiteSorter (tested with a) EmptyTestCase".
tests/Runner/TestSuiteSorterTest.php
Outdated
| $sorter = new TestSuiteSorter(); | ||
| $sorter->setCache($cache); | ||
| $_sorter = new \ReflectionClass($sorter); |
There was a problem hiding this comment.
_? leading _?
what about $sorterReflection ?
also, having a need to access class' internals in test indicate bad class structure. maybe Cache shall be injected to Sorter via constructor ?
There was a problem hiding this comment.
The $_var as a temporary version of $var as I quickly needed one, like the $_suite in TestRunner. Will pick something else for now, then look at the use of Reflection.
I can imagine $sorter = new TestSuiteSorter($cache) but to me that feels like juggling the hot potato of persisting the cache between runs. When added to the constructor it becomes the problem of the TestRunner which creates the TestSuiteSorter.
Thank you for your detailed comments! I will spend more time experimenting with the code and structure later today.
src/Runner/TestSuiteSorter.php
Outdated
| @@ -51,7 +51,7 @@ | |||
| ]; | |||
|
|
|||
| /** | |||
| * @var array | |||
| * @var array[string]int Associative array of (string => DEFECT_SORT_WEIGHT) elements | |||
There was a problem hiding this comment.
array[string]int is not proper type according to phpdoc
array<string, int>
There was a problem hiding this comment.
Thanks! Where did you find this syntax? I saw this recommended a few times, but have found no definitive answer in the manual https://docs.phpdoc.org/guides/types.html
I found some discussions regarding proposals but most of them seem to have gone to sleep, for example: phpDocumentor/phpDocumentor#650
There was a problem hiding this comment.
psr-5 always-proposal, sadly. but it's widely suported by IDEs and SCAs
src/Util/TestResultCache.php
Outdated
| return; | ||
| } | ||
|
|
||
| $this->loadFromJSON($json); | ||
| $cache = \unserialize($cacheData); |
There was a problem hiding this comment.
please provide 2nd argument with allowed class deserialization.
now, you are exposing injection point for security hole (for PHPUnit it's not big deal, but still...)
There was a problem hiding this comment.
Yes thx :) had added that in the meantime. Since you're here, is it very picky about namespacing?
There was a problem hiding this comment.
self::class takes FQCN into consideration, so all good ;)
tests/Util/TestResultCacheTest.php
Outdated
| public function testCacheFilenameViaEnv() | ||
| { | ||
| $_ENV['PHPUNIT_RESULT_CACHE'] = '/some/cache'; | ||
| $cache = new TestResultCache; | ||
|
|
||
| $this->assertEquals('/some/cache', $cache->getResultCacheFilename()); | ||
| unset($_ENV['PHPUNIT_RESULT_CACHE']); |
There was a problem hiding this comment.
if assertEquals would fail,
this line would never be executed
| private $runState = []; | ||
|
|
||
| /** | ||
| * @var TestResultCache |
There was a problem hiding this comment.
new ResultCacheListener()
boom ! this property is null, not TestResultCache ;)
There was a problem hiding this comment.
added some phpdoc ...which I then even had to sort 🤓
src/Runner/TestSuiteSorter.php
Outdated
| /** | ||
| * @var TestResultCache | ||
| */ | ||
| private $cache; |
There was a problem hiding this comment.
still.
new TestSuiteSorter()and this property is null, not TestResultCache
use ?TestResultCache or null|TestResultCache
There was a problem hiding this comment.
There might be a few more of those, I will check and change my code style for this.
|
@keradus Hope your offline breakfast was good :) as mentioned in the chat, here's the pull request for the defects-first project. Any comments very welcome! Small things I will take care of ASAP, larger changes to functionality for e.g. mutation testing for a next version. |
|
Goodmorning @sebastianbergmann! The functionality and code for Breakfast are done, so let's stick a fork in it. ;-) I have been using the changes for a few days now and it looks solid. I've checked off the list of changes you requested after I made the PR and taken care of @keradus suggestions so far.
While working on this a lot of usage scenarios have come up and I have plenty of ideas to make this more useful in the future. I have started to look at other PHP quality and testing projects such as Infection and the various scripts, listeners and printers users have created over the years. Thanks again for all your work on PHPUnit! Hope my added functionality is useful and saves somebody somewhere a bit of time and frustration. :) |
|
Sounds great ;) |
|
@epdenouden Thank you for letting me know, I hope to find the time to look at this ASAP. In the meantime, I look forward to see @keradus' review :) Thanks, Dariusz! |
| /** | ||
| * @var null|TestResultCache | ||
| */ | ||
| private $cache; |
There was a problem hiding this comment.
this is always passed via constructor now, it shall never be null
| final class ResultCacheExtension implements BeforeFirstTestHook, BeforeTestHook, AfterSuccessfulTestHook, AfterSkippedTestHook, AfterRiskyTestHook, AfterIncompleteTestHook, AfterTestErrorHook, AfterTestWarningHook, AfterTestFailureHook, AfterLastTestHook | ||
| { | ||
| /** | ||
| * @var array<string, array> array to keep track of suite/test state and timing information |
There was a problem hiding this comment.
what is the key? it's a string, that represents.... ?
what is the value? array of.... ?
maybe some example would be nice to let us know what's inside
There was a problem hiding this comment.
By using the $time provided to the individual tests I was able to refactor the $runstate out completely. The previous ResultCacheListener also kept track of TestSuite timings, which was done using this internal run state.
| $testname = $this->getTestName($test); | ||
|
|
||
| if (!isset($this->runState[$testname])) { | ||
| $this->runState[$testname] = [ |
There was a problem hiding this comment.
imo, it's a big shortcut to redirect all hooks here.
For example, it's not responsibility of executeAfterTestWarning to set start time, it shall not be in need to even check condition here.
Same for other hooks and other actions
There was a problem hiding this comment.
I have simplified the ResultCacheExtension quite a lot. It directly tells the cache when it has something of interest, bypassing the need for the internal state and a few hooks.
src/Framework/TestResult.php
Outdated
| $this->stop(); | ||
| } | ||
| } elseif ($t instanceof SkippedTest) { | ||
| $this->skipped[] = new TestFailure($test, $t); | ||
| $notifyMethod = 'addSkippedTest'; | ||
|
|
||
| if ($this->stopOnSkipped) { | ||
| if ($this->stopOnSkipped || $this->stopOnDefect) { |
There was a problem hiding this comment.
to be honest not sure about this...
basically, it means that I will never use --stopOnDefect
in bigger projects, there are always some tests depending on environment, eg different version of PHP, OS, ext, ini...
There was a problem hiding this comment.
Yes, thanks for noticing! This is also a huge doubt of mine. On the one hand I want to have a shorthand for "I just want a quick edit-test-edit-test cycle right now", but it has also gotten in my way so much I had moved the "not really bad" defects down the sorter list.
For now I will remove the halting part and leave it up to the user to supply explicit --stop-on-warning, etc options
There was a problem hiding this comment.
please dont
(elaboration soon)
There was a problem hiding this comment.
stopping on given type of result (~defect) shall work the same as putting them to cache to be executed first.
it's great value for me to have --stop-on-defect rather than --stop-on-* 10 times.
Only thing is to properly specify what is a defect.
For me, it all that "executed and failed" - thus all that need to manual fix from developer.
Thus, not met expectation, warning, ..., but not "skipped" or "incomplete"
There was a problem hiding this comment.
I like this suggestion very much: the first version I made was configurable. Remember that code style fix for the defectsWeights constant? That was a leftover of the original configurable version which I thought overcomplicated at the time.
I will look for a nice way to add it back in.
There was a problem hiding this comment.
Rereading your comment: things that need a manual fix from the developer in the test or source would be:
- error, warning: usually PHP syntax mistakes
- failure: assert failing
- risky: developer forgot to add assertions to a new test
Am I reading you correctly?
There was a problem hiding this comment.
exactly.
to clarify - i'm not thinking about making it configurable again, it shall not.
defects are all that require developer fix, so exactly ones you listed 👍
src/TextUI/Command.php
Outdated
| @@ -1118,9 +1143,11 @@ protected function showHelp(): void | |||
| --printer <printer> TestListener implementation to use | |||
|
|
|||
| --resolve-dependencies Resolve dependencies between tests | |||
| --defects-first-order Run previously unsuccessful tests first | |||
| --random-order Run tests in random order | |||
| --random-order-seed=<N> Use a specific random seed <N> for random order | |||
| --reverse-order Run tests last-to-first | |||
There was a problem hiding this comment.
what's the outcome of setting up both, --reverse-order and --defects-first-order ? will it be respected, ignored, crashed ?
There was a problem hiding this comment.
looks like --defects-first-order overrides --random-order and --reverse-order
There was a problem hiding this comment.
It actually combines them. Each of the sorters has a specific preference and some are more important than others:
- random and reverse go first, they are considered mostly cosmetic. In theory they have no effect on the outcome.
- More important is sorting the defects to the front. It is an explicit request from the (human) user that indicates "I have seen what happens during the runs and I would like an explicit change to that."
- The dependency resolver has the last say since its job is preventing breakages. A side effect of defects first I found is getting unrelated failures as the defects were sometimes sorted the wrong 'side' of their dependants.
There was a problem hiding this comment.
I agree I need to explicitly document this behaviour and am working on a few diagrams that show the limits of the sorters when looking at how suites can be inside other suites, what dependencies really do, and so on.
There was a problem hiding this comment.
if you need big description and diagrams, it's probably means it's overcomplicated.
maybe let us shrink it down to what would be the most value for user?
I would suggest sth like:
--order-by=default|reverse|random|defect
without possibility to mix them, like reverse + defect. I don't see much value of that combination, do you ?
There was a problem hiding this comment.
Yes, I think that would be better. My personal use case of 'randomized but with previous defects first' might be too exotic. Let's keep it simple.
There was a problem hiding this comment.
@sebastianbergmann do you have a preference. Guessing from your other comments, you prefer the simpler UX as well?
src/TextUI/TestRunner.php
Outdated
| $arguments['backupStaticAttributes'] = $arguments['backupStaticAttributes'] ?? null; | ||
| $arguments['beStrictAboutChangesToGlobalState'] = $arguments['beStrictAboutChangesToGlobalState'] ?? null; | ||
| $arguments['beStrictAboutResourceUsageDuringSmallTests'] = $arguments['beStrictAboutResourceUsageDuringSmallTests'] ?? false; | ||
| $arguments['cacheResult'] = $arguments['cacheResult'] ?? false; |
There was a problem hiding this comment.
shouldn't this be enabled by default? this is super useful feature, why to hide it ?
There was a problem hiding this comment.
I agree! It's in my local config file. :) I'll leave that up to Sebastian if he wants it on by default.
There was a problem hiding this comment.
maybe, for that, let us enable this for now in PR, and see Sebastian's overall review ?
There was a problem hiding this comment.
Alright let's go
src/Util/Configuration.php
Outdated
|
|
||
| default: | ||
| $result['executionOrderDefects'] = TestSuiteSorter::ORDER_DEFAULT; | ||
|
|
There was a problem hiding this comment.
dummy lines doesn't look needed here
There was a problem hiding this comment.
This supports using the config attribute executionOrderDefects='default' which can happen when automated pipelines do something like executionOrderDefects='$value'. I have added a test for it.
There was a problem hiding this comment.
I meant the empty ("dummy") line number 960
There was a problem hiding this comment.
I thought that was way too simple! Even your comments on whitespace scare me into making more tests haha. Removed btw
| * | ||
| * <code> | ||
| * // Record running time for test | ||
| * $this->times[$testName] = 1.234; |
src/Util/TestResultCache.php
Outdated
| /** | ||
| * Alternative path and filename for result cache file | ||
| * | ||
| * @var null|string |
There was a problem hiding this comment.
when could this be null, if it fallbacks to DEFAULT_RESULT_CACHE_FILENAME ?
There was a problem hiding this comment.
Good catch, cleaned up. In previous versions null was used to the default as a fallback in another part of the code.
| $suite->run($this->result); | ||
|
|
||
| $this->assertEquals(BaseTestRunner::STATUS_WARNING, $this->cache->getState('Warning')); | ||
| $this->assertEquals(0, $this->cache->getTime('test_name')); |
There was a problem hiding this comment.
assertSame please. globally
There was a problem hiding this comment.
Missed a few, fixed!
tests/Util/TestResultCacheTest.php
Outdated
| $cache->load(); | ||
|
|
||
| $this->assertEquals(BaseTestRunner::STATUS_UNKNOWN, $cache->getState('testOne')); | ||
| $this->assertEquals(BaseTestRunner::STATUS_SKIPPED, $cache->getState('testFive')); |
There was a problem hiding this comment.
assertSame please, globally
| { | ||
| /** | ||
| * @var array<string, array> array to keep track of suite/test state and timing information | ||
| */ | ||
| private $runState = []; |
There was a problem hiding this comment.
ping @epdenouden ,
dropping this property will save a lot of memory for huge testbase (like 15023 of mine), good job 👍
| @@ -68,6 +68,8 @@ class Command | |||
| protected $longOptions = [ | |||
| 'atleast-version=' => null, | |||
| 'bootstrap=' => null, | |||
| 'cache-result' => null, | |||
| 'cache-result-file=' => null, | |||
There was a problem hiding this comment.
I am not sure if we really want this to be configurable using CLI options. If caching could only be turned on through phpunit.xml then the hidden cache file could be placed next to it.
There was a problem hiding this comment.
If caching could only be turned on through
phpunit.xml
cache-result is to enable cache via CLI
There was a problem hiding this comment.
I understand that. What I am not sure about is the value of being able to turn on caching on the commandline.
There was a problem hiding this comment.
Hmmm yes. I haven't really turned it off anymore since it became stable to use. 🤔
The cache-result-file I would leave in, have found it to come in handy when scripting, like -c <config>.
| @@ -505,6 +519,11 @@ protected function handleArguments(array $argv): void | |||
|
|
|||
| break; | |||
|
|
|||
| case '--stop-on-defect': | |||
There was a problem hiding this comment.
I do not understand why --stop-on-defect is needed.
There was a problem hiding this comment.
Strictly it isn't. I added it as a shorthand for a combination of --stop-on-$status flags. @keradus do you have a more compelling explanation?
There was a problem hiding this comment.
With current definition of "defect", it's nothing more than a shortcut for putting down --stop-on-* multiple times.
It's not mandatory to have, but it's super nice shortcut for fast fail, eg for CLI.
(a lot of users runs all tests in local env, but want to fail on first issue on CI level)
So, I agree it's not needed. It's just to enhance UX.
| @@ -1133,6 +1160,7 @@ protected function showHelp(): void | |||
| --include-path <path(s)> Prepend PHP's include_path with given path(s) | |||
| -d key[=value] Sets a php.ini value | |||
| --generate-configuration Generate configuration file with suggested settings | |||
| --cache-result-file==<FILE> Specify result cache path and filename | |||
There was a problem hiding this comment.
What about "Specify path to result cache file"?
There was a problem hiding this comment.
It contains a the full $path . $filename not just a path. The filename itself doesn't have to be .phpunit.result.cache
src/TextUI/TestRunner.php
Outdated
| $arguments['timeoutForLargeTests'] = $arguments['timeoutForLargeTests'] ?? 60; | ||
| $arguments['timeoutForMediumTests'] = $arguments['timeoutForMediumTests'] ?? 10; | ||
| $arguments['timeoutForSmallTests'] = $arguments['timeoutForSmallTests'] ?? 1; | ||
| $arguments['verbose'] = $arguments['verbose'] ?? false; |
There was a problem hiding this comment.
I do not understand this whitespace change.
There was a problem hiding this comment.
Me neither, to be honest. I have just removed all whitespace between ] and = and had it reformatted. Looks like the assignment alignment doesn't remove excess whitespace once it has been introduced. Will commit a WS fix
|
@keradus @sebastianbergmann In 11360b3 I have added a suggestion for what a cleaner
Notes:
|
src/TextUI/Command.php
Outdated
| --defects-first-order Run previously unsuccessful tests first | ||
| --random-order Run tests in random order | ||
| --order-by=<order> Run tests in 'default', 'random' or 'reverse' order | ||
| --order-by=defects Run previously failed tests first |
There was a problem hiding this comment.
not sure about this docs here. it's single option, it shall have single doc entry, in which we explain what is defects
There was a problem hiding this comment.
also, looks like depends is missing
There was a problem hiding this comment.
Good points, yes. Wanted to give it a quick try using it myself for a while first. If this is something that @sebastianbergmann also likes I will make it production-ready.
There was a problem hiding this comment.
It barely fits on one line now, but everybody reads the manual... right? :)
|
|
||
| private function handleOrderByOption(string $value): void | ||
| { | ||
| foreach (\explode(',', $value) as $order) { |
There was a problem hiding this comment.
if I would put as value foobar, it shall fail. now, it's ignored
There was a problem hiding this comment.
Yes. I didn't see other --options throw nice error messages yesterday. Will have a look at how to do this nicely. Silent failures are a big no-no here, I have typed defecst waaaaay too often
There was a problem hiding this comment.
Added a simple error message with failure exit code.
|
@keradus The The help message is now: |
src/TextUI/Command.php
Outdated
| @@ -1347,6 +1346,8 @@ private function handleOrderByOption(string $value): void | |||
| $this->arguments['resolveDependencies'] = true; | |||
|
|
|||
| break; | |||
| default: | |||
| $this->exitWithErrorMessage("unrecognized order-by option -- $order"); | |||
There was a problem hiding this comment.
Fixed. Was a leftover from where I found this in GetOpt.
phpunit.xsd
Outdated
| @@ -171,6 +171,12 @@ | |||
| <xs:enumeration value="random"/> | |||
| </xs:restriction> | |||
| </xs:simpleType> | |||
| <xs:simpleType name="executionOrderDefectsType"> | |||
There was a problem hiding this comment.
this shall be removed now, and executionOrderType shall be extended to contain defects and depends
There was a problem hiding this comment.
Removed. I have added a list of the common scenarios that make sense. Couldn't find a nice way to define the list in the XSD to help the tooling with validation and autocomplete.
src/Framework/TestResult.php
Outdated
| @@ -1011,6 +1016,14 @@ public function stopOnSkipped(bool $flag): void | |||
| $this->stopOnSkipped = $flag; | |||
| } | |||
|
|
|||
| /** | |||
| * Enables or disables the stopping for any defect: skipped, error, failure, warning, incomplete or risky. | |||
There was a problem hiding this comment.
I believe this description is not up to date now
| 'testFour' => ['state' => BaseTestRunner::STATUS_FAILURE, 'time' => 1], | ||
| 'testFive' => ['state' => BaseTestRunner::STATUS_FAILURE, 'time' => 1], | ||
| ], | ||
| ['testFive', 'testOne', 'testTwo', 'testThree', 'testFour']], |
There was a problem hiding this comment.
👎
testFour and testFive failed, so I want to have them as soon as possible, with respect to their dependencies.
I would expect that order, instead:
['testFive', 'testThree', 'testFour', 'testOne', 'testTwo']
or
['testThree', 'testFour', 'testFive', 'testOne', 'testTwo']
as in the end, we claimed to run defects first, not last (vide testFour)
There was a problem hiding this comment.
testThreehas dependencies ontestOneandtestTwotestFourhas a dependency ontestThree
what happens:
- after sorting for defects the order is:
4(failed),5(failed),1,2,3 - running it like this would skip
4because it needs to run3first - same story for
3: it needs both1and2 - the dependency resolver fixes this quietly for us:
5can stay up front (=defects FIRST) but4goes behind3which goes behind1,2(=defects ASAP ;-)
There was a problem hiding this comment.
ha, ok, your first 2 points were not part of description (while dependencies of testThree for TestFour was)
then, all good 👍
There was a problem hiding this comment.
Good but not good enough :) I will document the use of MultiDependencyTest in the dataproviders to avoid some of the confusion.
src/TextUI/TestRunner.php
Outdated
| $arguments['processIsolation'] = $arguments['processIsolation'] ?? false; | ||
| $arguments['processUncoveredFilesFromWhitelist'] = $arguments['processUncoveredFilesFromWhitelist'] ?? false; | ||
| $arguments['randomOrderSeed'] = $arguments['randomOrderSeed'] ?? \time(); | ||
| $arguments['registerMockObjectsFromTestArgumentsRecursively']= $arguments['registerMockObjectsFromTestArgumentsRecursively'] ?? false; |
There was a problem hiding this comment.
a space is missing before =
after solving that, alignment will be back to original one again
There was a problem hiding this comment.
I tried something like that in the past, let's see if your suggestion works nicer.
|
@sebastianbergmann Hi Sebastian! @keradus and I think this pull request is done. For new ideas and improvements I will open new PRs in the near future. :) |
|
Thanks! |
|
🎁 thanks for the merge! :) |
|
🎆 |
Jis sugeneruojamas, kad būtų greitesnis testavimo-koregavimo ciklas. Įjungtas standatiškai nuo PHPUnit 8 Dokumentacija: * sebastianbergmann/phpunit#3147 * https://phpunit.de/announcements/phpunit-8.html
Jis sugeneruojamas, kad būtų greitesnis testavimo-koregavimo ciklas. Įjungtas standatiškai nuo PHPUnit 8 Dokumentacija: * sebastianbergmann/phpunit#3147 * https://phpunit.de/announcements/phpunit-8.html
Use case
Rerun defective unit tests first to speed up the red-green-refactor cycle!
Test results and timings are shared between tests runs using a cache. This allows for sorting defective tests to the front of the execution order using an additional sorter. This new sort option is compatible with existing test ordering options.
This project is a followup to my previous work on reordering test execution. The code is available in my Breakfast development branch.
Summary of changes
TestSuiteSortergains the ability to sort suites and tests by priority of defects and execution timeResultCacheExtensionto gather result and timing information during test runsTestResultCachefor sharing of result state between runs--cache-resultand configuration attributecacheResult='true'--order-by=defectsand configuration attributeexecutionOrder='defects'--cache-result-fileand configuration attributecacheResultFileHow to test
To activate storing the results of a run use
--cache-result:For this example we switch the cache on for every run by adding the attribute
cacheResult="true"to the<phpunit>element inphpunit.xml.Break
TestCaseTestby running it backwards:Look at the cache file. By default this is
.phpunit.result.cachein the PHPUnit working directory.Break
TestCaseTestagain with the sorting feature turning on:The skipped tests are still there but are run immediately now, but still fail. This is easily avoided by asking the sorter to keep dependencies in mind:
Design considerations
Like the New Order feature the aim is for a robust system with sane defaults that silently does its work and is easily maintained by other developers:
cacheResult="true"to your configuration and it silently does its work.