"No tests executed" does not always result in exit code 1

[remicollet]

And exit with an error code.

If you run phpunit, this means you very probably have tests.

This can avoid such issue: egulias/EmailValidator#244
as for now travis stay "green" while there is obviously something wrong.

Perhaps in next 9.3 or 10.0
Perhaps with a configuration option, with default=no in next and default=yes in next++

Open for discussion.

[sebastianbergmann]

This already exists.

<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;

final class Test extends TestCase
{
}

$ phpunit Test.php
PHPUnit 9.2.3 by Sebastian Bergmann and contributors.

W                                                                   1 / 1 (100%)

Time: 00:00.012, Memory: 6.00 MB

There was 1 warning:

1) Warning
No tests found in class "Test".

WARNINGS!
Tests: 1, Assertions: 0, Warnings: 1.

$ echo $?                     
0

$ phpunit --fail-on-warning Test.php
PHPUnit 9.2.3 by Sebastian Bergmann and contributors.

W                                                                   1 / 1 (100%)

Time: 00:00.014, Memory: 6.00 MB

There was 1 warning:

1) Warning
No tests found in class "Test".

WARNINGS!
Tests: 1, Assertions: 0, Warnings: 1.

$ echo $?                           
1

[remicollet]

Nah, not the same case.

$ mkdir foo
$ phpunit9 --fail-on-warning  foo
PHPUnit 9.2.3 by Sebastian Bergmann and contributors.

Time: 00:00.001, Memory: 4.00 MB

No tests executed!

$ echo $?
0

[sebastianbergmann]

Thank you, @remicollet.

[remicollet]

To summarize the need, in linked project:

Tests directory was rename to tests but phpunit.xml was not updated.

[remicollet]

thanks

[localheinz]

Hmm, not sure - why would this be an error?

Imagine you set up a project using a skeleton, but do not have tests at the time. Why should the build fail when there are no tests?

Updating from phpunit/phpunit:9.2.3 to phpunit/phpunit:9.2.4 now effectively fails all builds for projects executing empty test suites.

[sebastianbergmann]

The reasoning is this: PHPUnit was asked to run tests. You expect tests to be run. No tests were run. This does not match your expectation.

[localheinz]

It did for me - I have somewhere between 20 and 30 repositories where I have

• auto-review
• unit
• integration

test suites, all configured with separate phpunit.xml configuration files.

Not all of these test suites have tests in every repository, which has worked fine. All of these repositories use the same or similar workflow definitions, so I can easily keep workflows - whether intended for GitHub Actions or other build tools - synchronized.

With this change, I now need to go through all of the repositories making use of phpunit/phpunit:^9 and remove empty test suites and adjust workflows to make the builds pass.

While subtle, I would consider this a breaking change.

[localheinz]

Perhaps we can introduce an option that allows failing with an error when a test suite fails is empty?

Similar to how infection/infection allows to run without failure when no mutations were created?

[sebastianbergmann]

PHPUnit behaved exactly like this for a long time. Then at some point this behaviour was unintentionally changed. I was not aware of this until @remicollet brought this to my attention.

I think that PHPUnit should behave as it does right now. I understand (now, thank you, @localheinz) that fixing this after it was broken for so long is a break of backward compatibility. I shall revert the change now.

[localheinz]

Thank you, @sebastianbergmann!

I'll be happy to provide an option to fail for empty test suites - what do you think?

[sebastianbergmann]

That will be the topic of a ticket that I will open soon (after I have rolled back the change).

[sebastianbergmann]

For reference: #4313