TestListener object argument loader regression

[llaville]

Q	A
PHPUnit version	9.5.8
PHP version	7.4.22
Installation Method	PHAR distrib from official site https://phar.phpunit.de/

Summary

I've detected a regression on PHPUnit 9.5 when we use an object/class as argument of a testlistener

Current behavior

Class 'MyLoggerObjArg' not found

How to reproduce

Here are code I used to reproduce issue

source code

examples/boostrap.php

<?php declare(strict_types=1);

require dirname(__DIR__) . '/vendor/autoload.php';

class MyLoggerObjArg
{
}

class MyLogger implements \PHPUnit\Framework\TestListener
{
    private $target;

    public function __construct($class = null)
    {
        $this->target = __DIR__ . '/testlistener-phpunit-' .  \PHPUnit\Runner\Version::series() . '-' . date('YmdHis') . '.log';

        $data = \PHPUnit\Runner\Version::getVersionString() . PHP_EOL;
        error_log($data, 3, $this->target);

        if (is_object($class)) {
            error_log('class exists(My Logger Object Arg) ? ' . (class_exists('MyLoggerObjArg') ? 'yes': 'no') . PHP_EOL,3, $this->target);
        }
    }

    public function addError(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addWarning(\PHPUnit\Framework\Test $test, \PHPUnit\Framework\Warning $e, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addFailure(\PHPUnit\Framework\Test $test, \PHPUnit\Framework\AssertionFailedError $e, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addIncompleteTest(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addRiskyTest(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function addSkippedTest(\PHPUnit\Framework\Test $test, Throwable $t, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function startTestSuite(\PHPUnit\Framework\TestSuite $suite): void
    {
        $data = __FUNCTION__ . ', ' . $suite->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function endTestSuite(\PHPUnit\Framework\TestSuite $suite): void
    {
        $data = __FUNCTION__ . ', ' . $suite->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function startTest(\PHPUnit\Framework\Test $test): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }

    public function endTest(\PHPUnit\Framework\Test $test, float $time): void
    {
        $data = __FUNCTION__ . ', ' . $test->getName() . PHP_EOL;
        error_log($data, 3, $this->target);
    }
}

examples/testSuite.php

<?php declare(strict_types=1);

namespace Your\Name_Space;

use PHPUnit\Framework\TestCase;

class YourTestSuite extends TestCase
{
    public function testIncomplete()
    {
        // Optional: Test anything here, if you want.
        $this->assertTrue(TRUE, 'This should already work.');

        // Stop here and mark this test as incomplete.
        $this->markTestIncomplete(
          'This test has not been implemented yet.'
        );
    }

    public function testRisky()
    {
    }

    public function testSkipped()
    {
        $this->markTestSkipped('This test was skipped for any reason.');
    }

    public function testFailure()
    {
        $this->assertEmpty(array('foo'));
    }

    public function testPass()
    {
        $this->assertTrue(TRUE, 'This should always work.');
    }

    /**
     * @dataProvider additionProvider
     */
    public function testDataProvider($a, $b, $expected)
    {
        $this->assertEquals($expected, $a + $b);
    }

    public function additionProvider()
    {
        return [
            [0, 0, 0],
            [0, 1, 1],
            [1, 0, 1],
            [1, 1, 3]
        ];
    }
}

examples/listener_no_arg.xml

<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
         backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap.php"
         colors="true"
         stopOnError="false"
         stopOnFailure="false"
         stopOnIncomplete="false"
         stopOnRisky="false"
         stopOnSkipped="false"
         beStrictAboutTestsThatDoNotTestAnything="true"
         verbose="false">
    <listeners>
        <listener class="MyLogger">
        </listener>
    </listeners>
    <testsuites>
        <testsuite name="Demo Test Suite">
            <file>testSuite.php</file>
        </testsuite>
    </testsuites>
</phpunit>

examples/listener_obj_arg.xml

<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
         backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap.php"
         colors="true"
         stopOnError="false"
         stopOnFailure="false"
         stopOnIncomplete="false"
         stopOnRisky="false"
         stopOnSkipped="false"
         beStrictAboutTestsThatDoNotTestAnything="true"
         verbose="false">
    <listeners>
        <listener class="MyLogger">
            <arguments>
                <object class="MyLoggerObjArg" />
            </arguments>
        </listener>
    </listeners>
    <testsuites>
        <testsuite name="Demo Test Suite">
            <file>testSuite.php</file>
        </testsuite>
    </testsuites>
</phpunit>

WARNING I've noticed also that a startTestSuite end endTestSuite with no entry are generated only with PHPUnit 9.5,
while it was not on PHPUnit 8.5 and even PHPUnit 7.5

See

output

PHPUnit 9.5.8 by Sebastian Bergmann and contributors.
startTestSuite, 
startTestSuite, Demo Test Suite
startTestSuite, Your\Name_Space\YourTestSuite
startTest, testIncomplete
addIncompleteTest, testIncomplete
endTest, testIncomplete
startTest, testRisky
addRiskyTest, testRisky
endTest, testRisky
startTest, testSkipped
addSkippedTest, testSkipped
endTest, testSkipped
startTest, testFailure
addFailure, testFailure
endTest, testFailure
startTest, testPass
endTest, testPass
startTestSuite, Your\Name_Space\YourTestSuite::testDataProvider
startTest, testDataProvider with data set #0
endTest, testDataProvider with data set #0
startTest, testDataProvider with data set #1
endTest, testDataProvider with data set #1
startTest, testDataProvider with data set #2
endTest, testDataProvider with data set #2
startTest, testDataProvider with data set #3
addFailure, testDataProvider with data set #3
endTest, testDataProvider with data set #3
endTestSuite, Your\Name_Space\YourTestSuite::testDataProvider
endTestSuite, Your\Name_Space\YourTestSuite
endTestSuite, Demo Test Suite
endTestSuite, 

Expected behavior

With following commands: phpunit -c examples/listener_no_arg.xml or even phpunit -c examples/listener_obj_arg.xml

output

PHPUnit 9.5.8 by Sebastian Bergmann and contributors.

IRSF....F                                                           9 / 9 (100%)

Time: 00:00.004, Memory: 18.00 MB

There were 2 failures:

1) Your\Name_Space\YourTestSuite::testFailure
Failed asserting that an array is empty.

/shared/backups/bartlett/phpunit-bootstrap/examples/testSuite.php:31

2) Your\Name_Space\YourTestSuite::testDataProvider with data set #3 (1, 1, 3)
Failed asserting that 2 matches expected 3.

/shared/backups/bartlett/phpunit-bootstrap/examples/testSuite.php:44

--

There was 1 risky test:

1) Your\Name_Space\YourTestSuite::testRisky
This test did not perform any assertions

/shared/backups/bartlett/phpunit-bootstrap/examples/testSuite.php:20

FAILURES!
Tests: 9, Assertions: 7, Failures: 2, Skipped: 1, Incomplete: 1, Risky: 1.

[sebastianbergmann]

Although you provide a lot of information, I am sorry to say that I do not understand what you are reporting.

[llaville]

Well, in summary, when I run the command phpunit -c examples/listener_obj_arg.xml I expect to have the Expected Behavior output provided.
It's OK on PHPUnit 7.5 and PHPUnit, 8.5 but it's impossible on PHPUnit 9.x

I got Class 'MyLoggerObjArg' not found

This is the argument used as parameter of the class listener

Hope my new explains are better !

PS: if you want to see a real implementation of concept, check on my project https://github.com/llaville/phpunit-LoggerTestListener (branch master was recently updated to allow PHPUnit 8.x and PHPUnit9)
The corresponding examples/listener_obj_arg.xml is https://github.com/llaville/phpunit-LoggerTestListener/blob/master/examples/phpunit.monolog.xml#L18

[sebastianbergmann]

I have to admit that debugging and fixing issues related to test listeners is not very high on my list. Test listeners have been deprecated for years and will be removed in PHPUnit 10. I will probably not work on this issue, but would review/accept a pull request that fixes your problem.

[llaville]

I don't want to hurt you, but I'm a bit sad to hear that it won't be fixed.

At the base, the request to support PHPUnit 9.x come to a user of the community of my project https://github.com/llaville/phpunit-LoggerTestListener

When I tried to add support to PHPUnit 9, I've begun by a POC on Test Hooks, just before reading the essential post :
#4676
Thanks for it to clarify the situation about Test Hooks and Listeners.

Finally, I won't add support to PHPUnit 9 and release a new version of my project that will at least support PHPUnit 7.5 and PHPUnit 8.5
that do the job efficiently.

After all, PHPUnit 8 is supposed to support as PHPUnit 9 both PHP 7.3 and PHP 7.4 (https://phpunit.de/supported-versions.html)

Thanks to spent time to reply, I'd appreciated a lot !

[sebastianbergmann]

I briefly looked at it and the root cause appears to be that we're trying to create the object before the autoloader, which is also configured in the XML configuration file, has been loaded. I will try to find time to work on this.

[sebastianbergmann]

Anything related to TestListener in PHPUnit 8.5 and PHPUnit 9.5 will not be changed anymore and has been removed in PHPUnit 10.