TestCase::expectError works on PHP 7.3 but not on PHP 7.4/8.0

[boesing]

Q	A
PHPUnit version	9.5.4
PHP version	7.4.16, 7.3.27, 8.0.3
Installation Method	Composer

Summary

Hey there, I have a unit test which verifies if an error is being thrown. I am using TestCase::expectError for this, which actually works when executing unit tests with PHP 7.3.
When executing unit tests with PHP 7.4 or PHP 8.0, the test fails.

Current behavior

Failed asserting that exception of type "Error" matches expected exception "PHPUnit\Framework\Error\Error".

How to reproduce

<?php
declare(strict_types=1);


use PHPUnit\Framework\TestCase;

final class FooTest extends TestCase
{
    public function testError(): void
    {
        $this->expectError();
        join([new stdClass]);
    }
}

PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

F                                                                   1 / 1 (100%)

Time: 00:00.019, Memory: 6.00 MB

There was 1 failure:

1) FooTest::testError
Failed asserting that exception of type "Error" matches expected exception "PHPUnit\Framework\Error\Error". Message was: "Object of class stdClass could not be converted to string" at FooTest.php:12
.

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

Expected behavior

I'd expect that the unit test will pass on all PHP versions (7.3, 7.4 & 8.0).

PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

.                                                                   1 / 1 (100%)

Time: 00:00.004, Memory: 6.00 MB

OK (1 test, 1 assertion)

[sebastianbergmann]

Thank you for bringing this to my attention. There seems to have been a change to how error handlers work and/or how errors are raised between PHP 7.3 and PHP 7.4 that I was not aware of so far.

Here is a minimal reproducing example extracted from PHPUnit:

test.php

<?php declare(strict_types=1);
namespace test;

ini_set('error_reporting', '-1');

$errorHandler = new ErrorHandler(true, true, true, true);
$errorHandler->register();

try {
    $a = $b;
} catch (\Throwable $t) {
    var_dump(get_class($t));
}

try {
    \implode([new \stdClass]);
} catch (\Throwable $t) {
    var_dump(get_class($t));
}

final class ErrorHandler
{
    /**
     * @var bool
     */
    private $convertDeprecationsToExceptions;

    /**
     * @var bool
     */
    private $convertErrorsToExceptions;

    /**
     * @var bool
     */
    private $convertNoticesToExceptions;

    /**
     * @var bool
     */
    private $convertWarningsToExceptions;

    /**
     * @var bool
     */
    private $registered = false;

    public function __construct(bool $convertDeprecationsToExceptions, bool $convertErrorsToExceptions, bool $convertNoticesToExceptions, bool $convertWarningsToExceptions)
    {
        $this->convertDeprecationsToExceptions = $convertDeprecationsToExceptions;
        $this->convertErrorsToExceptions       = $convertErrorsToExceptions;
        $this->convertNoticesToExceptions      = $convertNoticesToExceptions;
        $this->convertWarningsToExceptions     = $convertWarningsToExceptions;
    }

    public function __invoke(int $errorNumber, string $errorString, string $errorFile, int $errorLine): bool
    {
        /*
         * Do not raise an exception when the error suppression operator (@) was used.
         *
         * @see https://github.com/sebastianbergmann/phpunit/issues/3739
         */
        if (!($errorNumber & error_reporting())) {
            return false;
        }

        switch ($errorNumber) {
            case E_NOTICE:
            case E_USER_NOTICE:
            case E_STRICT:
                if (!$this->convertNoticesToExceptions) {
                    return false;
                }

                throw new Notice($errorString, $errorNumber, $errorFile, $errorLine);

            case E_WARNING:
            case E_USER_WARNING:
                if (!$this->convertWarningsToExceptions) {
                    return false;
                }

                throw new Warning($errorString, $errorNumber, $errorFile, $errorLine);

            case E_DEPRECATED:
            case E_USER_DEPRECATED:
                if (!$this->convertDeprecationsToExceptions) {
                    return false;
                }

                throw new Deprecated($errorString, $errorNumber, $errorFile, $errorLine);

            default:
                if (!$this->convertErrorsToExceptions) {
                    return false;
                }

                throw new Error($errorString, $errorNumber, $errorFile, $errorLine);
        }
    }

    public function register(): void
    {
        if ($this->registered) {
            return;
        }

        $oldErrorHandler = set_error_handler($this);

        if ($oldErrorHandler !== null) {
            restore_error_handler();

            return;
        }

        $this->registered = true;
    }

    public function unregister(): void
    {
        if (!$this->registered) {
            return;
        }

        restore_error_handler();
    }
}

class Error extends \Exception
{
    public function __construct(string $message, int $code, string $file, int $line, \Exception $previous = null)
    {
        parent::__construct($message, $code, $previous);

        $this->file = $file;
        $this->line = $line;
    }
}

final class Deprecated extends Error
{
}

final class Notice extends Error
{
}

final class Warning extends Error
{
}

PHP 7.3

string(11) "test\Notice"
string(10) "test\Error"

PHP 7.4

string(11) "test\Notice"
string(5) "Error"

PHP 8.0

string(12) "test\Warning"
string(5) "Error"

[sebastianbergmann]

There seems to have been a change to how error handlers work and/or how errors are raised between PHP 7.3 and PHP 7.4 that I was not aware of so far.

Nothing changed in general, but specifically to how (string) new stdClass is treated: it was changed to throw an Error in PHP 7.4 where it was a "recoverable fatal error" before.

[boesing]

Thanks for the insights Sebastian!
I've changed my code around and throw a RuntimeException instead.
This should prevent regressions in my library even tho, PHP might change that error again to something else 👍🏼