Skip to content

phpunit/phpunit-selenium does not work with PHPUnit 8.5.9 #4516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bobemoe opened this issue Nov 12, 2020 · 2 comments
Closed

phpunit/phpunit-selenium does not work with PHPUnit 8.5.9 #4516

bobemoe opened this issue Nov 12, 2020 · 2 comments
Assignees
@sebastianbergmann
Labels
type/bug Something is broken

Comments

@bobemoe
Copy link

bobemoe commented Nov 12, 2020

Summary

After update from 8.5.8 to 8.5.9 phpunit seems to be "testing" all the internal methods of the Selenium2TestCase.

Current behaviour

A simple Selenium2TestCase test fails, with hundreds of failed tests.

PHPUnit 8.5.9 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.3.19-1~deb10u1

Unnamed Tests (App\Tests\UnnamedTests)
 ☢ Set up [1022.11 ms]
   │
   │ This test did not perform any assertions
   │
   │ /home/bob/Projects/phpunittest/test.php:8
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ✔  [1350.83 ms]
 ✘ Share session [1183.80 ms]
   │
   │ ArgumentCountError: Too few arguments to function PHPUnit\Extensions\Selenium2TestCase::shareSession(), 0 passed in /home/bob/Projects/phpunittest/vendor/phpunit/phpunit/src/Framework/TestCase.php on line 1415 and exactly 1 expected
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:199
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:395
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ✘ Keep session on failure [1149.94 ms]
   │
   │ ArgumentCountError: Too few arguments to function PHPUnit\Extensions\Selenium2TestCase::keepSessionOnFailure(), 0 passed in /home/bob/Projects/phpunittest/vendor/phpunit/phpunit/src/Framework/TestCase.php on line 1415 and exactly 1 expected
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:213
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:395
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ✘ Session strategy [1730.11 ms]
   │
   │ BadMethodCallException: The command 'sessionStrategy' is not existent or not supported yet.
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase/CommandsHolder.php:159
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase/CommandsHolder.php:105
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:440
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:395
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ✘ Default session strategy [1119.84 ms]
   │
   │ BadMethodCallException: The command 'defaultSessionStrategy' is not existent or not supported yet.
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase/CommandsHolder.php:159
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase/CommandsHolder.php:105
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:440
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:395
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ☢ Default wait until timeout [1086.11 ms]
   │
   │ This test did not perform any assertions
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:240
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ✘ Set default wait until timeout [1100.62 ms]
   │
   │ ArgumentCountError: Too few arguments to function PHPUnit\Extensions\Selenium2TestCase::setDefaultWaitUntilTimeout(), 0 passed in /home/bob/Projects/phpunittest/vendor/phpunit/phpunit/src/Framework/TestCase.php on line 1415 and exactly 1 expected
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:248
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:395
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ☢ Default wait until sleep interval [1215.09 ms]
   │
   │ This test did not perform any assertions
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:257
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │

 ✘ Set default wait until sleep interval [1176.35 ms]
   │
   │ ArgumentCountError: Too few arguments to function PHPUnit\Extensions\Selenium2TestCase::setDefaultWaitUntilSleepInterval(), 0 passed in /home/bob/Projects/phpunittest/vendor/phpunit/phpunit/src/Framework/TestCase.php on line 1415 and exactly 1 expected
   │
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:265
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:395
   │ /home/bob/Projects/phpunittest/vendor/phpunit/phpunit-selenium/PHPUnit/Extensions/Selenium2TestCase.php:360
   │
^C

How to reproduce

Clone my simple test case

git clone https://github.com/bobemoe/phpunit_selenium_test
cd phpunit_selenium_test
composer install
vendor/bin/phpunit -v test.php --testdox

See here one test is ran, as expected. Now update phpunit and try again:

composer require phpunit/phpunit:8.5.9 --update-with-dependencies
vendor/bin/phpunit -v test.php --testdox

Now there are hundreds of tests being executed that aren't even defined in test.php.

Expected behaviour

The single test defined in test.php is executed.

More details at: https://github.com/bobemoe/phpunit_selenium_test

Composer show

doctrine/instantiator              1.4.0   A small, lightweight utility to instantiate objects in PHP without invoking their constructors
myclabs/deep-copy                  1.10.1  Create deep copies (clones) of your objects
phar-io/manifest                   1.0.3   Component for reading phar.io manifest information from a PHP Archive (PHAR)
phar-io/version                    2.0.1   Library for handling version information and constraints
phpdocumentor/reflection-common    2.2.0   Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock  5.2.2   With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information t...
phpdocumentor/type-resolver        1.4.0   A PSR-5 based resolver of Class names, Types and Structural Element Names
phpspec/prophecy                   1.12.1  Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage          7.0.10  Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator          2.0.2   FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template          1.2.1   Simple template engine.
phpunit/php-timer                  2.1.2   Utility class for timing
phpunit/php-token-stream           3.1.1   Wrapper around PHP's tokenizer extension.
phpunit/phpunit                    8.5.9   The PHP Unit Testing framework.
phpunit/phpunit-selenium           8.0.0   Selenium Server integration for PHPUnit
sebastian/code-unit-reverse-lookup 1.0.1   Looks up which function or method a line of code belongs to
sebastian/comparator               3.0.2   Provides the functionality to compare PHP values for equality
sebastian/diff                     3.0.2   Diff implementation
sebastian/environment              4.2.3   Provides functionality to handle HHVM/PHP environments
sebastian/exporter                 3.1.2   Provides the functionality to export PHP variables for visualization
sebastian/global-state             3.0.0   Snapshotting of global state
sebastian/object-enumerator        3.0.3   Traverses array structures and object graphs to enumerate all referenced objects
sebastian/object-reflector         1.1.1   Allows reflection of object attributes, including inherited and non-public ones
sebastian/recursion-context        3.0.0   Provides functionality to recursively process PHP variables
sebastian/resource-operations      2.0.1   Provides a list of PHP built-in functions that operate on resources
sebastian/type                     1.1.3   Collection of value objects that represent the types of the PHP type system
sebastian/version                  2.0.1   Library that helps with managing the version number of Git-hosted PHP projects
symfony/polyfill-ctype             v1.20.0 Symfony polyfill for ctype functions
theseer/tokenizer                  1.2.0   A small library for converting tokenized PHP source code into XML and potentially other formats
webmozart/assert                   1.9.1   Assertions to validate method input/output with nice error messages.
@bobemoe bobemoe added the type/bug Something is broken label Nov 12, 2020
@bobemoe
Copy link
Author

bobemoe commented Nov 13, 2020

I've done a git bisect:

155e5b0549093e2c494cc303c2910daeac0b046c is the first bad commit
commit 155e5b0549093e2c494cc303c2910daeac0b046c
Author: Sebastian Bergmann <sb@sebastian-bergmann.de>
Date:   Sun Sep 27 17:14:46 2020 +0200

    Motivated by #4471, do not even try to add non-public methods as test methods

:040000 040000 4e1583ccc1a2664ecfb6e541e17ef6aaa7eafa1d ea04683349274931b66805ccbe1a9267dbea71ea M	src
:040000 040000 2b568eaa88abb9cbffc9e3be5f0853061d38b593 0d121c476be755b98ae26de705f799fea8947cf6 M	tests

@sebastianbergmann sebastianbergmann changed the title phpunit 8.5.9 update breaks Selenium2TestCase phpunit/phpunit-selenium does not work with PHPUnit 8.5.9 Nov 13, 2020
@sebastianbergmann sebastianbergmann self-assigned this Nov 13, 2020
@sebastianbergmann
Copy link
Owner

phpunit/phpunit-selenium uses PHPUnit\Framework\TestSuite although this class is marked as @internal and therefore must not be used outside of PHPUnit itself.

The correct fix would be for phpunit/phpunit-selenium to no longer unconditionally call addTestMethod() and instead ensure it only calls this method with a valid method.

That being said, I will revert the change in the 8.5 branch. However, I will not revert this change in the 9.4 and master branches.

CC @giorgiosironi

@bobemoe bobemoe mentioned this issue Feb 6, 2021
Closed
This was referenced Mar 12, 2021
Merged
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sebastianbergmann sebastianbergmann

Labels
type/bug Something is broken
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sebastianbergmann @bobemoe