JUnit logger reports warnings as failures

[celeryclub]

Q	A
PHPUnit version	8.3.4
PHP version	7.3.7
Installation Method	Composer

Hello there. I looks like JUnit output in PHPUnit 8.3.4 is incorrect. My test suite generates 1 warning (@expectedException deprecation) and all tests pass, but in the JUnit output it shows 1 failure.

Here is a minimal example reproducing the behavior: https://gist.github.com/celeryclub/389ac4d5089255730014d44b236a046e.

Thanks for looking and please let me know if any additional information would be helpful.

"composer info | sort" output:

doctrine/instantiator              1.2.0   A small, lightweight utility to instantiate objects in PHP without invoking their c...
myclabs/deep-copy                  1.9.3   Create deep copies (clones) of your objects
phar-io/manifest                   1.0.3   Component for reading phar.io manifest information from a PHP Archive (PHAR)
phar-io/version                    2.0.1   Library for handling version information and constraints
phpdocumentor/reflection-common    1.0.1   Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock  4.3.1   With this component, a library can provide support for annotations via DocBlocks or...
phpdocumentor/type-resolver        0.4.0
phpspec/prophecy                   1.8.1   Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage          7.0.7   Library that provides collection, processing, and rendering functionality for PHP c...
phpunit/php-file-iterator          2.0.2   FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template          1.2.1   Simple template engine.
phpunit/php-timer                  2.1.2   Utility class for timing
phpunit/php-token-stream           3.1.0   Wrapper around PHP's tokenizer extension.
phpunit/phpunit                    8.3.4   The PHP Unit Testing framework.
sebastian/code-unit-reverse-lookup 1.0.1   Looks up which function or method a line of code belongs to
sebastian/comparator               3.0.2   Provides the functionality to compare PHP values for equality
sebastian/diff                     3.0.2   Diff implementation
sebastian/environment              4.2.2   Provides functionality to handle HHVM/PHP environments
sebastian/exporter                 3.1.1   Provides the functionality to export PHP variables for visualization
sebastian/global-state             3.0.0   Snapshotting of global state
sebastian/object-enumerator        3.0.3   Traverses array structures and object graphs to enumerate all referenced objects
sebastian/object-reflector         1.1.1   Allows reflection of object attributes, including inherited and non-public ones
sebastian/recursion-context        3.0.0   Provides functionality to recursively process PHP variables
sebastian/resource-operations      2.0.1   Provides a list of PHP built-in functions that operate on resources
sebastian/type                     1.1.3   Collection of value objects that represent the types of the PHP type system
sebastian/version                  2.0.1   Library that helps with managing the version number of Git-hosted PHP projects
symfony/polyfill-ctype             v1.12.0 Symfony polyfill for ctype functions
theseer/tokenizer                  1.1.3   A small library for converting tokenized PHP source code into XML and potentially o...
webmozart/assert                   1.4.0   Assertions to validate method input/output with nice error messages.

[sebastianbergmann]

Your example is missing the PHPUnit XML configuration file you use.

[celeryclub]

Hey, thanks for looking. I'm not using an XML config file for this example project - I'm just running the test script in composer.json - it does this: ./vendor/bin/phpunit --bootstrap vendor/autoload.php --log-junit junit.xml test/DemoTest.php

[DanielRuf]

Hi @celeryclub,

see #3832.

You can test it with the following patch:
https://github.com/sebastianbergmann/phpunit/pull/3832.patch or https://github.com/sebastianbergmann/phpunit/pull/3832.diff

This should properly add the warnings attribute.

[sebastianbergmann]

"JUnit output in PHPUnit 8.3.4 is incorrect" suggests that previous versions of PHPUnit behaved differently. I do not think that this is the case.

I wonder whether it would make sense to introduce a configuration setting for the JUnit logfile writer that controls whether or not warnings are logged. If they are, they should -- IMO -- be logged as errors.

And, of course, this touches upon #3162.

[epdenouden]

The correct/incorrect depends on who you ask. A toggle would be nice and have the current version of PHPUnit stick to the existing behaviour.

[sebastianbergmann]

What @epdenouden says does not sound wrong :)

[celeryclub]

Hey, sorry for the delay. I was off the grid for a bit. I apologize if the title was misleading - I only meant that I had identified this behavior in 8.4.3. I think you're probably correct that it's been around for a while.

@DanielRuf I tried your patch - it does behave the way I'd expect. Thanks for that!

@epdenouden I suppose however you look at it, it would be safer to not introduce a breaking change. If we were to add a toggle, do you imagine that being something that would go into phpunit.xml?