Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing_app hangs on Windows #1000

Open
johnpryan opened this issue Jan 20, 2022 · 6 comments
Open

testing_app hangs on Windows #1000

johnpryan opened this issue Jan 20, 2022 · 6 comments
Labels
p2

Comments

@johnpryan
Copy link
Contributor

After enabling Flutter desktop on Windows, running flutter test integration_test -d windows hangs until ctrl+c, then a TimeoutException is printed.

@dnfield @csells I followed the setup instructions at https://docs.flutter.dev/desktop and I can successfully run the app under test (flutter run -d windows)

The integration tests in this sample are currently failing on a macOS device: (#999)

Logs 
C:\Users\johnp\code\samples\testing_app>flutter test integration_test -d windows
00:00 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart                                         B21:49 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart [E]
  TimeoutException after 0:12:00.000000: Test timed out after 12 minutes.
  package:test_api/src/backend/invoker.dart 333:28  Invoker._handleError.<fn>
  dart:async/zone.dart 1418:47                      _rootRun
  dart:async/zone.dart 1328:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 331:10  Invoker._handleError
  package:test_api/src/backend/invoker.dart 287:9   Invoker.heartbeat.<fn>.<fn>
  dart:async/zone.dart 1426:13                      _rootRun
  dart:async/zone.dart 1328:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 286:38  Invoker.heartbeat.<fn>
  dart:async-patch/timer_patch.dart 18:15           Timer._createTimer.<fn>
  dart:isolate-patch/timer_impl.dart 395:19         _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 426:5          _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 192:12      _RawReceivePortImpl._handleMessage

21:49 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart
21:50 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart                                     Terminate batch job (Y/N)? y

C:\Users\johnp\code\samples\testing_app>flutter --version
Flutter 2.10.0-0.1.pre • channel beta • https://github.com/flutter/flutter.git
Framework • revision 628f0e3f3a (9 days ago) • 2022-01-11 19:50:05 -0600
Engine • revision 63ca99584a
Tools • Dart 2.16.0 (build 2.16.0-134.1.beta) • DevTools 2.9.2
@csells
Copy link

csells commented Jan 20, 2022

@dnfield can you take a look at this? I'd love integration_test to work well on Windows.

@johnpryan
Copy link
Contributor Author

After running the tests again, I can see one of the tests passing but another gets a TestDeviceException:

logs 
C:\Users\johnp\code\samples\testing_app>flutter test integration_test -d windows
00:00 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart                                         B00:16 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart
00:26 +4: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart                                        B00:40 +4: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\state_mgmt_test.dart                               B00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart [E]
  Bad state: Cannot add new events after calling close
  dart:async/broadcast_stream_controller.dart 243:24  _BroadcastStreamController.add
  dart:async/zone.dart 1434:47                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 2314:41             _Socket._onData
  dart:async/zone.dart 1442:13                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 1839:33             new _RawSocket.<fn>
  dart:io-patch/socket_patch.dart 1322:14             _NativeSocket.issueReadEvent.issue
  dart:async/schedule_microtask.dart 40:21            _microtaskLoop
  dart:async/schedule_microtask.dart 49:5             _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 122:13        _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 193:5         _RawReceivePortImpl._handleMessage

C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: The command "setlocal [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -DBUILD_TYPE=Debug -P cmake_install.cmake [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmErrorLevel [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: exit /b %1 [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmDone [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :VCEnd" exited with code 1. [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
Building Windows application...
00:55 +4 -2: loading C:\Users\johnp\code\samples\testing_app\integration_test\state_mgmt_test.dart [E]
  Exception: Build process failed.
  package:flutter_tools/src/base/common.dart 10:3                    throwToolExit
  package:flutter_tools/src/windows/build_windows.dart 311:5         _runBuild
  ===== asynchronous gap ===========================
  package:flutter_tools/src/windows/build_windows.dart 77:5          buildWindows
  ===== asynchronous gap ===========================
  package:flutter_tools/src/windows/windows_device.dart 62:5         WindowsDevice.buildForDevice
  ===== asynchronous gap ===========================
  package:flutter_tools/src/desktop_device.dart 123:7                DesktopDevice.startApp
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/integration_test_device.dart 53:39  IntegrationTestTestDevice.start

00:55 +4 -2: Some tests failed.

C:\Users\johnp\code\samples\testing_app>

@johnpryan
Copy link
Contributor Author

It looks like there's some sort of race condition when testing more than one file at a time... I can consistently run flutter test integration_test\app_test.dart -d windows and flutter test integration_test\perf_test.dart -d windows, but not flutter test integration_test -d windows.

@dnfield
Copy link
Contributor

dnfield commented Jan 20, 2022

@cbracken might be better set up to help with this (or know who is). My access to Windows machines is fairly limited

@cbracken cbracken self-assigned this Jan 20, 2022
@cbracken
Copy link
Member

Thanks for the heads-up. Taking a look.

@miquelbeltran
Copy link
Member

miquelbeltran commented Jan 25, 2022 •
edited by RedBrogdon

@csells requested me to put here my findings.

Setup

flutter doctor 
$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, 2.10.0-0.2.pre, on Microsoft Windows [Version 10.0.22000.434], locale en-150)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
[✓] Chrome - develop for the web
[✓] Visual Studio - develop for Windows (Visual Studio Community 2019 16.11.4)
[✓] Android Studio (version 2020.3)
[✓] VS Code (version 1.63.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

• No issues found!

All tests running on Windows 11 with the Git Bash on Windows Terminal.

Results

Running flutter test integration_test -d windows:

  • the app_test.dart runs and passes
  • the perf_test.dart eventually fails with the following error (see logs)
  • the state_mgmt_test.dart doesn't get to run, also fails (app is a weird state)
logs 
$ flutter test integration_test -d windows
Running "flutter pub get" in testing_app...                      2,280ms
00:00 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart                                  B00:26 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart
00:37 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart                                 B00:50 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart                        B00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart [E]
  Bad state: Cannot add new events after calling close
  dart:async/broadcast_stream_controller.dart 243:24  _BroadcastStreamController.add
  dart:async/zone.dart 1434:47                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 2314:41             _Socket._onData
  dart:async/zone.dart 1442:13                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 1839:33             new _RawSocket.<fn>
  dart:io-patch/socket_patch.dart 1322:14             _NativeSocket.issueReadEvent.issue
  dart:async/schedule_microtask.dart 40:21            _microtaskLoop
  dart:async/schedule_microtask.dart 49:5             _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 122:13        _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 193:5         _RawReceivePortImpl._handleMessage


Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
01:05 +4 -2: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

01:05 +4 -2: Some tests failed.

Running flutter test integration_test/app_test.dart -d windows:

  • test runs and passes
  • command never finishes (I had to press q to finish it)
logs 
$ flutter run integration_test/app_test.dart -d windows
Launching integration_test/app_test.dart on Windows in debug mode...
Building Windows application...
flutter: 00:00 +0: Testing App Driver Tests Finding an item in the list
Syncing files to device Windows...                                  53ms

Flutter run key commands.
r Hot reload. 🔥🔥🔥
R Hot restart.
h List all available interactive commands.
d Detach (terminate "flutter run" but leave application running).
c Clear the screen
q Quit (terminate the application on the device).

💪 Running with sound null safety 💪

An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:63630/YJ2FEDpVaPI=/
The Flutter DevTools debugger and profiler on Windows is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:63630/YJ2FEDpVaPI=/
flutter: 00:00 +1: Testing App Driver Tests Testing IconButtons
flutter: 00:05 +2: Testing App Driver Tests Verifying whether item gets added to favorites
flutter: 00:06 +3: Testing App Driver Tests Testing remove button
flutter: 00:10 +4: Testing App Driver Tests (tearDownAll)
flutter: Warning: integration_test plugin was not detected.
flutter:
flutter: If you're running the tests with `flutter drive`, please make sure your tests
flutter: are in the `integration_test/` directory of your package and use
flutter: `flutter test $path_to_test` to run it instead.
flutter:
flutter: If you're running the tests with Android instrumentation or XCTest, this means
flutter: that you are not capturing test results properly! See the following link for
flutter: how to set up the integration_test plugin:
flutter:
flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
flutter:
flutter: 00:10 +5: All tests passed!

Application finished.

Running performance test:

  • Run the command flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart -d windows
  • Runs and finishes successfully.
logs 
$ flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart -d windows
Running "flutter pub get" in testing_app...                        990ms
Building Windows application...
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:63762/X8EGYWluyoE=/
VMServiceFlutterDriver: Isolate found with number: 706933644293459
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: Testing App Performance Tests Scrolling test
VMServiceFlutterDriver: Connected to Flutter application.
flutter: 00:04 +1: Testing App Performance Tests Favorites operations test
flutter: 00:15 +2: Testing App Performance Tests (tearDownAll)
flutter: Warning: integration_test plugin was not detected.
flutter:
flutter: If you're running the tests with `flutter drive`, please make sure your tests
flutter: are in the `integration_test/` directory of your package and use
flutter: `flutter test $path_to_test` to run it instead.
flutter:
flutter: If you're running the tests with Android instrumentation or XCTest, this means
flutter: that you are not capturing test results properly! See the following link for
flutter: how to set up the integration_test plugin:
flutter:
flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
flutter:
flutter: 00:15 +3: All tests passed!
All tests passed.

Running state_mgmt_test.dart:

  • Same result as with the app_test.dart:
  • Runs successfuly
  • the testing app remains open, needs to be closed with q
logs 
$ flutter run integration_test/state_mgmt_test.dart -d windows
Launching integration_test/state_mgmt_test.dart on Windows in debug mode...
Building Windows application...
flutter: 00:00 +0: Testing App State Management Tests Verifying add method
Syncing files to device Windows...                                  55ms

Flutter run key commands.
r Hot reload. 🔥🔥🔥
R Hot restart.
h List all available interactive commands.
d Detach (terminate "flutter run" but leave application running).
c Clear the screen
q Quit (terminate the application on the device).

💪 Running with sound null safety 💪

An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:63842/agkZOurH5FE=/
The Flutter DevTools debugger and profiler on Windows is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:63842/agkZOurH5FE=/
flutter: 00:00 +1: Testing App State Management Tests Verifying remove method
flutter: 00:00 +2: Testing App State Management Tests (tearDownAll)
flutter: Warning: integration_test plugin was not detected.
flutter:
flutter: If you're running the tests with `flutter drive`, please make sure your tests
flutter: are in the `integration_test/` directory of your package and use
flutter: `flutter test $path_to_test` to run it instead.
flutter:
flutter: If you're running the tests with Android instrumentation or XCTest, this means
flutter: that you are not capturing test results properly! See the following link for
flutter: how to set up the integration_test plugin:
flutter:
flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
flutter:
flutter: 00:00 +3: All tests passed!

Application finished.

Removing the files drive.dart perf_driver.dart and perf_test.dart from the integration_test folder, then running flutter test integration_test -d windows:

  • The test app_test.dart runs and passes.
  • The test state_mgmt_test.dart starts and fails.
  • A blank app window appears on the screen and remains open.
logs 
$ flutter run integration_test -d windows
Target file "integration_test" not found.

Miquel@MIGUEL-WINDOWS MINGW64 ~/Documents/dev/samples/testing_app (master)
$ flutter test integration_test -d windows
00:00 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart                                  B00:13 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart
00:24 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart                           B00:37 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
00:37 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

00:37 +4 -1: Some tests failed.

Summary

  • It seems that when more than one test file is in the integration_test folder, the command flutter test integration_test will fail on the second file.
  • Running a single test leaves the command running and needs to be closed with q (or control+c).
  • The fact that the performance tests are in the same folder seems unrelated to the root cause of this issue.

@cbracken cbracken mentioned this issue Apr 26, 2022
Open
@domesticmouse domesticmouse changed the title testing_app hangs on Windows testing_app hangs on Windows Sep 18, 2023
sku0x20 added a commit to sku0x20/flutter_spike_state_management that referenced this issue Oct 29, 2023
@sku0x20
completed; issue with windows test runner, cannot run multiple integr…
5e7b4bc 
…ation tests.

anyways, that was a long way. this was in such a bad shape. learned things, again. still so much.
but with this, state idea i got. ui i need to see. ui is never ending. but now i have an idea how to approach and can always look back here for insights. quick look for further lookup. starting idea. and 'this exists'.
still this has multiple issues. like that delay of 8secs. idk, i have no idea how to solve that rn. fine for now tbh.
we will see.

i liked working with windows.
and how just easily it is to add support for android or windows to existing project. flutter create --platforms=android .

but in any case i liked this spike. learned quite good. was doing stuff badly last time. wow.
still so much to learn.

flutter/flutter#101031
flutter/samples#1000
jtdLab/rapid#114
@cbracken cbracken removed their assignment Mar 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
p2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@domesticmouse @cbracken @johnpryan @miquelbeltran @csells @dnfield