GitHub Actions CI tests are failing on PHP 7.x

[osma]

At which URL did you encounter the problem?

https://github.com/NatLibFi/Skosmos/runs/5493589722

What steps will reproduce the problem?

1. Do something (e.g. open a PR) that triggers a GitHub Actions CI job

What is the expected output? What do you see instead?

Expected for PHPUnit tests to pass, but they are all failing on PHP 7.x with this kind of error:

 There were 482 errors:

1) BreadcrumbTest::testConstruct
PHPUnit\Framework\Exception: Fatal error: Uncaught TypeError: Argument 3 passed to PHP_Token::__construct() must be an instance of PHP_Token_Stream, instance of PHPUnit\PHP_Token_Stream given, called in phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php on line 131 and defined in /app/vendor/phpunit/php-token-stream/src/Token.php:37
Stack trace:
#0 phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php(131): PHP_Token->__construct('<?php\n', 1, Object(PHPUnit\PHP_Token_Stream), 0)
#1 phar:///usr/local/bin/phpunit/php-token-stream/Token/Stream.php(73): PHPUnit\PHP_Token_Stream->scan('<?php\n\n/**\n * A...')
#2 phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage.php(515): PHPUnit\PHP_Token_Stream->__construct('<?php\n\n/**\n * A...')
#3 phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage.php(496): PHPUnit\SebastianBergmann\CodeCoverage\CodeCoverage->getLinesToBeIgnoredInner('/app/model/Brea...')
#4 phar:///usr/local/bin/phpunit/php-code-coverage/CodeCoverage.php(450): PHPUnit\SebastianBergmann\CodeCoverage\CodeCo in /app/vendor/phpunit/php-token-stream/src/Token.php on line 37

PHP 8.0 is not affected - tests are working fine.

I just installed Skosmos on an Ubuntu 20.04 system with PHP 7.4 and ran the tests without problems. So the problem only appears under GitHub Actions CI currently.

[osma]

I tried debugging this. The first thing I did was to delete the Packages that php-actions has created in this repository, in order to force them to be rebuilt. However, that just made things worse as now the building of those packages fails, which means that tests are broken for PHP 8.0 as well, not just 7.x.

Another approach I tried was to fork the example-phpunit repo and test it in this branch: https://github.com/osma/example-phpunit/tree/test

But so far I haven't yet been able to replicate any of these problems outside the Skosmos repo. So for now, the tests are broken but there are no clues on how they could be fixed.

[osma]

The new problem (unable to recreate the Packages that php-build uses for caching) seems to be caused by missing permissions for GITHUB_TOKEN. The error message is denied: installation not allowed to Write organization package

This needs to be sorted out first, then we can see if the original issue (which only affects PHP 7.x) still remains.

[osma]

PR #1290 fixed the php-build issues, so now we are back at the starting point: PHP 7.x tests are failing for mysterious reasons on GitHub Actions, but tests pass on PHP 8.0.

[osma]

Downgrading to PHPUnit 8.5.23 (from 8.5.24) makes the tests work again on PHP 7.x.

[osma]

Digging deeper to identify the root cause...

I've cut down the Skosmos codebase and the test suite on the issue1288-debug-ci-tests branch to a bare minimum (basically just Breadcrumb.php and BreadcrumbTest.php) along with minimal Composer dependencies, disabled the autoloader etc. None of this changed the result; tests are still failing on PHP 7.x but not 8.0; even 8.1 started passing tests since the incompatible code got removed :)

But then this commit which disables the logging of coverage information by PHPUnit made the PHP 7.x tests work again. PHPUnit 8 doesn't support coverage on PHP 8.x anyway. So I strongly suspect this has to do with collecting coverage information.

[osma]

I was finally able to reproduce this locally, using PHAR archives of phpunit-8.5.23 and 8.5.24 and the PCOV extension installed. Luckily this seems to be unrelated to either php-build or the GitHub Actions CI environment.

Now I need to create a minimal test case and then report this on the phpunit issue tracker. I'll leave those for another day though.

[osma]

Reported as sebastianbergmann/phpunit#4934

[osma]

PHPUnit 8.5.25 with a fix to this issue was just released.
The CI jobs are now working again! 🎉
Closing this issue.

[kinow]

That was fast (both in PHPUnit and here in Skosmos 🏆 )