Remove AppVeyor, run E2E tests during CI #1365

sanmai · 2020-10-22T13:20:03Z

This PR:

Runs integration and end-to-end tests on Windows using GitHub Actions.
Adds a separate mutation testing workflow, with only PHP 7.4/PCOV and Linux/Windows
Disables AppVeyor (will report success all the time)
Runs full set of E2E tests on Ubuntu, without a .phar yet.

The one bit I cannot do: protected branch config needs new Windows checks to set required. Also, IIRC version branches are not protected, they might need some love too.

There's no need to drop a requirement for AppVeyor: it is still used by earlier branches.

- Use absolute path for PHPUnit. - Use bash as default shell in Makefile. - Run a subset of E2E tests on Windows. - Without BENCHMARK_SOURCES yet - Add a sane memory-limit - Update InitialConfigBuilderTest - Add a dedicated mutation testing workflow, with only PHP 7.4/PCOV and Linux/Windows

sanmai · 2020-10-23T01:03:03Z

Makefile

@@ -22,7 +23,7 @@ PHP_CS_FIXER_CACHE=build/cache/.php_cs.cache

 PHPSTAN=./vendor/bin/phpstan

-PHPUNIT=vendor/bin/phpunit
+PHPUNIT=vendor/phpunit/phpunit/phpunit


This path works both under Linux and on Windows.

sanmai · 2020-10-23T01:03:35Z

Makefile

@@ -1,3 +1,4 @@
+SHELL=bash


This isn't the default for Windows, therefore we have to be explicit here.

Makefile

sanmai · 2020-10-23T01:05:41Z

.github/workflows/ci.yaml

        run: |
-          php bin/infection -j2 --test-framework-options="--group=default" --ignore-msi-with-no-mutations
+          sudo apt-get install -y --no-install-recommends expect


Getting Expect on Windows, is, well, complicated.

This could be worked around with WSL, but it'll make Windows VM "unpure": we can't expect everyone using Infection with Windows with WSL.

sanmai · 2020-10-23T01:05:59Z

.github/workflows/ci.yaml

+        shell: bash
+        run: |
+          ls tests/e2e/*/composer.json | xargs dirname |
+             xargs -I{} composer --working-dir={} update --no-interaction --prefer-dist --no-progress ${{ matrix.dependencies }}


Surprise! This works on Windows too!

tests/phpunit/Logger/ConsoleLoggerTest.php

sanmai · 2020-10-23T08:54:42Z

.github/workflows/ci.yaml

+        if: runner.os == 'Windows'
+        shell: bash
+        run: |
+          make test-e2e-phpunit PHP='phpdbg -qrr' E2E_PHPUNIT_GROUP=e2e BENCHMARK_SOURCES=


It just wouldn't work with Xdebug. It seems installed, but then it is not.

sanmai · 2020-10-23T08:56:51Z

.github/workflows/autoreview.yaml

          restore-keys: |
-            composer-${{ runner.os }}-${{ matrix.php-version }}-${{ hashFiles('**/composer.*') }}-
+            composer-${{ runner.os }}-${{ matrix.php-version }}-${{ hashFiles('composer.*') }}-


We only need to consider two files here, no others in tests/e2e or otherwise.

sanmai · 2020-10-23T08:59:40Z

.github/workflows/ci.yaml

+            e2e-vendor-${{ runner.os }}-${{ matrix.php-version }}-${{ hashFiles('tests/e2e/*/composer.json') }}-
+            e2e-vendor-${{ runner.os }}-${{ matrix.php-version }}-
+            e2e-vendor-${{ runner.os }}-
+            composer-


Installing anything with Composer is insanely slow in Windows, just compare these 4 seconds on Linux with these 34 seconds on Windows. The solution is to cache the dependencies, but not only archives, the whole vendor folders. Note that we don't cache composer.lock here: it takes far less time to verify that all is in order.

This might be a mistake, but let's see.

sanmai · 2020-10-23T09:00:38Z

Makefile

@@ -39,6 +40,8 @@ BENCHMARK_SOURCES=tests/benchmark/MutationGenerator/sources \
 				  tests/benchmark/Tracing/coverage \
 				  tests/benchmark/Tracing/sources

+E2E_PHPUNIT_GROUP=integration,e2e
+PHPUNIT_GROUP=default


To let us override these from outside.

sanmai · 2020-10-23T09:02:13Z

tests/e2e/SymfonyFlex/run_tests.bash

+    fi
+
+    diff -u -w expected-output.txt infection.log
+}


SymfonyFlex needs a special environment variable, and it uses old version of PHPUnit which is not compatible with the one we use, so we can't really run this test from E2ETest. Adding this file prevents this test from running from E2ETest, and also solves env variable issue.

sanmai · 2020-10-23T09:02:36Z

tests/phpunit/TestFramework/PhpUnit/Config/Builder/InitialConfigBuilderTest.php

+        if (PHP_EOL === "\r\n") {
+            $this->markTestSkipped('Test fixture uses Unix line endings');
+        }
+


We haven't run this test on Windows before, apparently.

maks-rafalko · 2020-10-24T08:23:30Z

appveyor.yml

-
-matrix:
-  fast_finish: true
+clone_depth: 1


why do we need to keep this file?

AppVeyor will fail the build without it.

so will we remove once 0.18 is not maintained anymore?

Yes, that's the idea. Remove it once we don't need it anywhere.

tests/phpunit/TestFramework/PhpUnit/Config/Builder/InitialConfigBuilderTest.php

maks-rafalko · 2020-10-24T08:30:14Z

The one bit I cannot do: protected branch config needs new Windows checks to set required. Also, IIRC version branches are not protected, they might need some love too.

let's merge and I will set it up

- Use absolute path for PHPUnit. - Use bash as default shell in Makefile. - Run a subset of E2E tests on Windows. - Without BENCHMARK_SOURCES yet - Add a sane memory-limit - Update InitialConfigBuilderTest - Add a dedicated mutation testing workflow, with only PHP 7.4/PCOV and Linux/Windows

sanmai marked this pull request as ready for review October 22, 2020 14:25

sanmai force-pushed the pr/2020-10/no-appveyor branch from 3661bf2 to fcbca1a Compare October 23, 2020 01:21

sanmai added the DX Developer Experience label Oct 23, 2020

sanmai mentioned this pull request Oct 23, 2020

Improve Makefile, E2E tests #1371

Merged

6 tasks

sanmai commented Oct 23, 2020

View reviewed changes

sanmai force-pushed the pr/2020-10/no-appveyor branch from d868758 to 3d93ecf Compare October 23, 2020 06:02

sanmai commented Oct 23, 2020

View reviewed changes

sanmai requested a review from maks-rafalko October 23, 2020 09:02

sanmai mentioned this pull request Oct 23, 2020

Add github logger to be able to use Annotations on GitHub Actions #1368

Merged

2 tasks

sanmai force-pushed the pr/2020-10/no-appveyor branch from 7cd3ea8 to ade9fb4 Compare October 23, 2020 10:21

maks-rafalko approved these changes Oct 24, 2020

View reviewed changes

sanmai merged commit 4353f98 into infection:master Oct 24, 2020

sanmai deleted the pr/2020-10/no-appveyor branch October 24, 2020 08:33

theofidry mentioned this pull request Dec 21, 2022

refactor(ci): Review the mutation testing workflow #1791

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove AppVeyor, run E2E tests during CI #1365

Remove AppVeyor, run E2E tests during CI #1365

sanmai commented Oct 22, 2020 •

edited

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

sanmai Oct 23, 2020

maks-rafalko Oct 24, 2020

sanmai Oct 24, 2020

maks-rafalko Oct 24, 2020

sanmai Oct 24, 2020

maks-rafalko commented Oct 24, 2020

Remove AppVeyor, run E2E tests during CI #1365

Remove AppVeyor, run E2E tests during CI #1365

Conversation

sanmai commented Oct 22, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maks-rafalko commented Oct 24, 2020

sanmai commented Oct 22, 2020 •

edited