Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Behavior of fail-fast flag on 4.1.23? #6290

Closed
PZ01 opened this issue Dec 13, 2021 · 6 comments · Fixed by #6297
Closed

Behavior of fail-fast flag on 4.1.23? #6290

PZ01 opened this issue Dec 13, 2021 · 6 comments · Fixed by #6297

Comments

@PZ01
Copy link

PZ01 commented Dec 13, 2021

What are you trying to achieve?

Running tests inside of a Gitlab Pipeline. Expected to have folder specified in command line to be run only.

What do you get instead?

All tests ran, causing issues with our Pipeline

Provide console output if related. Use -vvv mode for more details.

echo "Running tests in folder ${CODECEPTION_TEST_FOLDER}"
Running tests in folder tests/unit
$ vendor/bin/codecept run --no-interaction --xml $CI_JOB_NAME.xml --fail-fast $CODECEPTION_TEST_FOLDER || EXIT_CODE=$?
Codeception PHP Testing Framework v4.1.23
Powered by PHPUnit 9.5.10 by Sebastian Bergmann and contributors.
[Seed] 242220195
Functional Tests (328) -------------------------------------
✖ PerformTest: It_should_return_something_specific (0.09s)

I saw fail-fast has been changed in 4.1.23, I set pipeline to 4.1.22 and it's working. Seems like if a default is not provided, it considers the path as a value to the flag or something? As you can see, the message 'Running tests in folder XXX' is actually running everything. I picked up on this because we have around 100 unit tests, and I saw that codecept started running everything(328).

It was my bad to keep ^4.0 in the pipeline, but I didn't expect the problem to be of this nature. --fail-fast without a number shouldn't be an impediment for folder path provided at the end.

Details

  • Codeception version: 4.1.23
  • PHP Version: 7.4
  • Operating System: Linux
  • Installation type: Composer
@b1rdex
Copy link
Contributor

b1rdex commented Dec 16, 2021

That's a breaking change introduced in #6275. We've got the problem too.

@b1rdex
Copy link
Contributor

b1rdex commented Dec 16, 2021

@Verest do you have any ideas how to solve this? Is it possible to check whether the value is an int (new behavior) or a path?

@Verest
Copy link
Contributor

Verest commented Dec 16, 2021

I looked and the section responsible for setting the arguments is within Symfony\Component\Console\Input\ArgvInput i.e. the third party. This is called within src/Codeception/Command/Run.php::execute() via $input->getOptions().

Symfony's documents state that a -- should be added after if not an option, I wasn't aware of this until now unfortunately. See: https://symfony.com/doc/current/components/console/console_arguments.html

From a cursory look, it seems php codecept run cli --fail-fast tests/cli/RunCest.php has the issue replicated here. php codecept run cli --fail-fast -- tests/cli/RunCest.php does fix it.

In retrospect, this shoulda been merged into develop as it does seem to be a breaking change. That said, I do not think this is a bug, rather expected behavior based on the Symfony docs.

@Verest
Copy link
Contributor

Verest commented Dec 16, 2021

I am running into issues adding unit tests themselves to test this. The executeCommand helper method would need to be updated as it is appending a -n to the end which will break any filepaths added onto the $command passed in. See tests/supportCliHelper.php.

@Naktibalda
Copy link
Member

I think that we have to revert fail-fast change in 4.1 branch.

@Naktibalda
Copy link
Member

Reverted in 4.1.26

@Verest I merged your feature to 5.0 branch, which should be released in a few weeks.

You can keep using 4.1.25 if you really need this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants