New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure mutated code is always valid #301
Conversation
0fe2318
to
3466d90
Compare
@@ -27,13 +27,19 @@ public function provideMutationCases(): \Generator | |||
<<<'PHP' | |||
<?php | |||
|
|||
(yield $a => $b); | |||
function test() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before this change, this mutator was failing with PHP Fatal error: The "yield" expression can only be used inside a function in /tmp/XXX
during linting
Because of the slowdowns IMO we should not do this in general and by default. By request and during CI - that's sure. Another way to lint is to use Scheme with fork could be even faster if we won't wait for a single process to finish, but rather collect all exit statuses later at some point. |
I'm not a big fan :/ While I do agree it would be better to keep generating valid code, there is two things I'm not fond of in that PR:
Maybe an alternative would be to try to be smarter in the way we generate the mutants? e.g. not removing a |
Here's a PoC of a forking linter: function eval_fork($code)
{
$pid = pcntl_fork();
if ($pid == 0) {
// Child
set_time_limit(1);
error_reporting(0);
eval($code);
exit();
}
if ($pid == -1) {
// fork() failed
return false;
}
// we are the parent
pcntl_wait($status);
return 0 == $status;
}
var_dump(eval_fork('$a = 1;')); // true
var_dump(eval_fork('$a = 1 /* ; */')); // false
$code = '<?php if (false) {class A extends B {}}';
var_dump(eval_fork('?>' . $code)); // true One call above takes approximately 11 ms. |
|
||
private function assertSyntaxIsValid(string $realMutatedCode) | ||
{ | ||
exec(sprintf('echo %s | php -l', escapeshellarg($realMutatedCode)), $output, $returnCode); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
proc_open
seems like a good fit here; there will be more LOC but less syscalls
I'd try with eval_fork
first, though.
@theofidry, unfortunately, I didn't get your comment at all.
I would like to highlight that this PR checks PHP code validity only in the PHPUnit Test Suite. Nothing has been changed in the infection itself. We don't lint Mutant code in the normal usage. This PR just double checks that Mutator generate the valid syntax in each Test Case in
What do you mean here?
That's right. And adding a test, that confirms mutated code is valid, is a great addition IMO @sanmai thanks for the suggestions, I will check it when I have a chance. From what I can see now, The only one concern here is that the code is evaluated (with |
We can no-op the code with var_dump(eval_fork('class A extends B {}')); // false
var_dump(eval_fork('if (false) {class A extends B {}}')); // true
var_dump(eval_fork('if (false) {yield true;}')); // false
var_dump(eval_fork('if (false) {function () {yield true;};}')); // true Doing |
function proc_open_linter($code) {
// We have to override stdout and stderr here,
// else they'll get connected to *our* stdin/stderr
// flooding them with errors
$process = proc_open('php -l', [
['pipe', 'r'],
['pipe', 'w'],
['pipe', 'r'],
], $pipes);
if (!is_resource($process)) {
return false;
}
// Pass our code
fwrite($pipes[0], $code);
fclose($pipes[0]);
// PHP won't write anything of interest into stderr,
// no point in looking there at:
// "Errors parsing Standard input code"
return proc_close($process) == 0;
}
var_dump(proc_open_linter('<?php return 1;')); // bool(true)
var_dump(proc_open_linter('<?php yield true;')); // bool(false)
var_dump(proc_open_linter('<?php var_dump(true); var_dump(true)')); // bool(false) I get about 40-50 ms per invocation |
@borNfreee nevermind my comment; I should have checked the diff :) |
@sanmai with I would stick to
Regarding |
Thank you @sanmai for your help |
This PR adds a check in unit tests that mutated code is always valid.
Extracted from #262 and this comment in particular
For reference:
Concerns about this PR:
With this new assertion we are sure that at least tested cases produce valid code for each Mutator.
Thoughts?