Disconnectedclient disconnected from CONNECTED state between every run in Angular

[stbth01]

This may be an Angular issue but I have not be able to find much except for similar issues here such as 1514 or 2971. However my issue seems different than those.

Expected behavior

Re-run test suite on change

Actual behavior

Although slow (different issues I suppose) the first round of tests are fine, but with singleRun set to false most changes to the code produce the testing error (shown below). Small changes, like minor html updates don’t always cause the disconnect. After making a change there is a long lag before either the error message pops up or the tests re-run, and if there is that error sometimes it will start back on its own but much of the time people on the team end up killing and starting over.

Environment Details

package.json
{
"name": "frontend",
"version": "0.0.0",
"scripts": {
"ng": "ng",
"start": "ng serve",
"build": "ng build",
"test": "ng test",
"coverage": "ng test --code-coverage",
"lint": "ng lint",
"e2e": "ng e2e",
"serve-site": "ng build && ng serve -o"
},
"private": true,
"dependencies": {
"@angular/animations": "^7.0.4",
"@angular/common": "^7.0.4",
"@angular/compiler": "^7.0.4",
"@angular/core": "^7.0.4",
"@angular/forms": "^7.0.4",
"@angular/http": "^7.0.4",
"@angular/platform-browser": "^7.0.4",
"@angular/platform-browser-dynamic": "^7.0.4",
"@angular/router": "^7.0.4",
"@ng-bootstrap/ng-bootstrap": "^4.0.0",
"auth0-js": "^9.8.2",
"bootstrap": "^4.1.3",
"classlist.js": "^1.1.20150312",
"core-js": "^2.5.4",
"ng2-nouislider": "^1.7.13",
"nouislider": "^12.1.0",
"rxjs": "~6.3.3",
"web-animations-js": "^2.3.1",
"zone.js": "~0.8.26"
},
"devDependencies": {
"@angular-devkit/build-angular": "^0.10.6",
"@angular/cli": "^7.0.6",
"@angular/compiler-cli": "^7.0.4",
"@angular/language-service": "^7.0.4",
"@fortawesome/fontawesome-free": "^5.5.0",
"@types/jasmine": "^2.8.12",
"@types/jasminewd2": "~2.0.3",
"@types/node": "~8.9.4",
"codelyzer": "~4.5.0",
"jasmine-core": "~2.99.1",
"jasmine-spec-reporter": "~4.2.1",
"karma": "~3.0.0",
"karma-chrome-launcher": "~2.2.0",
"karma-coverage-istanbul-reporter": "~2.0.1",
"karma-jasmine": "~1.1.2",
"karma-jasmine-html-reporter": "^0.2.2",
"protractor": "~5.4.0",
"ts-node": "~7.0.0",
"tslint": "~5.11.0",
"typescript": "~3.1.1"
}

Karma Config
{
basePath: '',
frameworks: ['jasmine', '@angular-devkit/build-angular'],
plugins: [
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-coverage-istanbul-reporter'),
require('@angular-devkit/build-angular/plugins/karma')
],
client: {
clearContext: false // leave Jasmine Spec Runner output visible in browser
},
coverageIstanbulReporter: {
dir: require('path').join(__dirname, '../coverage'),
reports: ['html', 'lcovonly'],
fixWebpackSourcePaths: true
},
reporters: ['progress', 'kjhtml'],
port: 9876,
colors: true,
logLevel: config.LOG_INFO,
autoWatch: true,
browsers: ['Chrome'],
singleRun: false,

captureTimeout: 210000,
browserDisconnectTolerance: 3,
browserDisconnectTimeout: 210000,
browserNoActivityTimeout: 210000,

}
}

Steps to reproduce the behaviour

1. npm test
2. make code change

[johnjbarton]

I guess you have to debug this by using chrome devtools. Somehow your test code or angular is interfering with karma-jasmine. Look in to adapter.js from that package. Also check that you are not interfering with the 'load' event or any other window event accidently in your test (by not mocking something eg). Check that you are not reloading the page.

I don't think this is something we can fix by any change in karma-runner.

[damianrebolo]

same here, Some knows where could be the problem?

[jvalkeejarvi]

For me disabling source maps helps. I get exactly the same error randomly when running tests with source maps on (which they are by default). When tests are run with command ng test --source-map=false there are no disconnect errors.

I suspect that vendor source maps are generated by default which makes source map files simply too large to load in sensible time.

There seems to be something wrong with sourceMap settings of test target. I created an issue to Angular CLI project :angular/angular-cli#13766

[Fatme]

I've reproduced the same error DisconnectedClient disconnected from CONNECTED state with the the following streps:

• Run ng test from terminal
• Close the browser after the tests are executed
• The above mentioned error is shown in the terminal

[johnjbarton]

That message means the browser is no longer connected, so if you close the browser, you should expect that message.

[jwedel]

I'm using headless chrome in IntelliJ and randomly (but in about 10-20% of test runs). This is quite annoying. When I rerun, most of the time s it works.

[johnjbarton]

The 'disconnected from CONNECTED' means that the browser socket was connected at one time in the past then disconnected. The tests either ran to completion or never started.

karma-runner has no direct control over the network or external processes. It's not like karma-runner can say "stop crashing" or "network don't fail me now".

The place where we could look for improvements is in the browser launchers: these monitor external processes and perhaps they can provide better feedback.

Which launcher are you using? And can you take a look into debugging the state of the process when this happens?

[jwedel]

The 'disconnected from CONNECTED' means that the browser socket was connected at one time in the past then disconnected. The tests either ran to completion or never started.

OK, so you're saying Karma is fine and Chrome has a bug? I mean, this is happens a lot and it would be great to fix this issue.

Which launcher are you using?

"karma-chrome-launcher": "~2.2.0"

And can you take a look into debugging the state of the process when this happens?

What exactly do you mean? Which process? Chrome, Node? And how to debug?

[johnjbarton]

I was imagining that one of the trace or log options in chrome might help:
https://peter.sh/experiments/chromium-command-line-switches/

You may be crashing the chrome tab due to say OOM but you are close to the limit so most of the time the tests finish before the memory exhausts.

Have you tried just disabling half of your tests to see if changes? Or split your test so only part of the app code has to be loaded?

(I don't think this is a bug in chrome or karma, but I'm open to ways to make the issue clearer).

[agoldh20]

This is a common issue at my company, nobody has found a work around yet

[damianrebolo]

Same at the company that I'm working.

[johnjbarton]

I just want to encourage you to think of this as a problem you are in a position to investigate. This does not happen to us even though we run karma on many thousands of tests across many browsers.

[jwedel]

@johnjbarton I understand, that it's hard to help when you cannot reproduce the issue. I tried to get some insights by adding

"--enable-tracing"
"--enable-tracing-output ./trace-output.log"

to the Karma config, but no file has been written. Could you provide a configuration that will produce sensible tracing output?

[johnjbarton]

The logging configuration options are documented in http://karma-runner.github.io/4.0/config/configuration-file.html

[agoldh20]

I changed my loglevel to this

        logLevel: config.LOG_DEBUG,
        loggers: [{type: 'console'}]

and this is the response I get from my console

[2019-09-04T11:40:18.909] [DEBUG] karma - All browsers are ready, executing
[2019-09-04T11:40:18.909] [DEBUG] karma - Captured 1 browsers
[2019-09-04T11:40:18.915] [WARN] HeadlessChrome 76.0.3809 (Mac OS X 10.14.6) - Disconnected (0 times)Client disconnected from CONNECTED state (ping timeout)
[2019-09-04T11:40:18.915] [DEBUG] HeadlessChrome 76.0.3809 (Mac OS X 10.14.6) - CONNECTED -> DISCONNECTED```

[jwedel]

@johnjbarton
Thanks, I think there has been a misunderstanding.
In your previous post, you gave me the command line options for chromium, the last post contains the configuration options for Karma.

As I statted before, I am happy to help investigating the issue further but my knowledge in Chrome and Karma internals is limited.

That's why I asked you, which Karma options should I set so that it would produce sensible debug output that helps you to reproduce it?

[johnjbarton]

That's why I asked you, which Karma options should I set so that it would produce sensible debug output that helps you to reproduce it?

In my experience, I don't believe any karma (or chrome) options will help here. I don't believe this issue can be fixed by karma (or chrome).

The most likely explanation for this problem is in your app or test code. That code is likely to be far more complex than karma, to be subject to fewer tests, and to be newer and change faster. In addition, the symptoms point in that direction.

Karma is launching the browser, instructing it to load the test page, and responding to the test page requests for test and app content. Then it is waiting for your test and app code to respond with test results. It never hears back. The simple explanation is that your code is exhausting some resource in the browser. As async code is likely involved, the outcome can be variable from run to run.

In --no-single-run karma is only doing one extra thing: not exiting. So when you reload it is just acting like a web server and giving your content to the browser. These 'disconnected' messages are just how karma says "Hey browser you are ignoring me, so ok I'm so done with you".

I would also consider removing these options:

captureTimeout: 210000,
browserDisconnectTolerance: 3,
browserDisconnectTimeout: 210000,
browserNoActivityTimeout: 210000,

These values are extraordinary. Maybe somewhat larger values than default may be needed in your case, but I would check that these are not masking an app or test issue.

[jwedel]

@johnjbarton Yes, we have a quite large application that takes a while to load. So this could probably be a reason. But I haven't seen the application / tab etc to crash on my machine. So this would more likely be a timeout issue than an out of memory issue unless the browser would behave differently when started by Karma.

Ah, and we don't have the above options set as @stbth01 that you suggested to remove.

[johnjbarton]

But I haven't seen the application / tab etc to crash on my machine.

What is it that you do see then? If karma disconnects then the browser has not responded. You should observe that the browser hangs during the test.

[jwedel]

I'm usually running Karme with Headless Chrome. What I meant is that when I run the application (not the tests) in Chrome, the tabs never crash although it takes some time to load.
I've also tried running Headed-Chrome some time ago and got the same issues as with Headless. The browser window stay open and show the Karma page with the red banner saying disconnected, as far as I can recall.

[johnjbarton]

Is the browser running on the same machine as karma?

[jwedel]

Yes

[agoldh20]

I'm usually running Karme with Headless Chrome. What I meant is that when I run the application (not the tests) in Chrome, the tabs never crash although it takes some time to load.
I've also tried running Headed-Chrome some time ago and got the same issues as with Headless. The browser window stay open and show the Karma page with the red banner saying disconnected, as far as I can recall.

I also only have issues when using Chromium or Headless Chrome.

[leonliang33]

I think I might have found the root cause of all these issues. What node version are you running?
I have two machines one on Node 8 and one on Node 10.
My machine with node 10 is hitting this disconnect issue pretty frequently.

[jwedel]

@leonliang33 I think I was using node 6 for quite some time (also having this issue). I switched to node 10 some days ago and still experiencing disconnects.

[johnjbarton]

All signs still point to app/test code here. I know you want this to be an infrastructure issue, but I think you are looking in the wrong direction. We run a lot of angular tests on node 10 + chrome.

[jwedel]

@johnjbarton I agree that I most likely depends on the application. I tried running tests a 100 times in a "tour of heroes" demo project without any issues. However, this demo app is quite small and has not that many elaborate tests.

Moreover, our application runs fine in prod environment. I've never seen it crashing the browser when opening it. I've never seen that issue happening on Jenkins either. It's only local.

You mentioned out-of-memory could be an issue.

I've also just noticed, that After exiting chrome (ui), there were like 10 chrome processes still running without any Browser or Karma servers running. Maybe it's really Chrome that is crashing somehow.

Is there any way to find traces of that in the logs?

[johnjbarton]

So this is an issue for only one environment? That suggested trying to isolate environment differences.

[jwedel]

@johnjbarton

• It appears on multiple developer laptops locally
• It appears not to happen on Jenkins
• We don't see general app crashes in production (this has no direct relation to Karma)

[johnjbarton]

This points to a race in test code. I guess by 'Jenkins' you mean CI tests on a cloud or cluster machine, typically lower power than desktops. So in that case one branch wins and the test passes. Locally the CPU/network is faster and a different branch wins the race and the test fails.

[jwedel]

This points to a race in test code. I guess by 'Jenkins' you mean CI tests on a cloud or cluster machine, typically lower power than desktops. So in that case one branch wins and the test passes. Locally the CPU/network is faster and a different branch wins the race and the test fails.

Yes, Jenkins is our CI build pipeline.

But do you mean by "tests fail" that the assertions fail? Because they don't. At least the test reporter doesn't even show that tests are running. I'll only get error message mentioned in the title.

[johnjbarton]

ok then "...different branch wins the race and the test page stops responding."

[jwedel]

@johnjbarton One thing I've noticed is, that with a running Karma server, I can immediately rerun tests when this error happens and then, the tests work. So I don't have to restart the Karma server. Not sure, is that in line with your assumption of the root cause?

[johnjbarton]

I can immediately rerun tests when this error happens

Please explain the steps you use to rerun.

[jwedel]

Sure!

• In IntelliJ, I start tests
• Karma Server Starts in Tool window
• second tool window opens that shows test results
• after some seconds, the disconnected error is shown
• click on rerun tests in tests tool window
• tests run fine

[johnjbarton]

second tool window opens that shows test results

This is pass or fail or error?

after some seconds, the disconnected error is shown

If this message did not appear, would everything else be ok?

click on rerun tests in tests tool window

? That is not a karma feature, can you explain it further? Is it IntelliJ? A button in the web page from the test framework?

[jwedel]

This is pass or fail or error?

Just the connection error, no pass/fail

If this message did not appear, would everything else be ok?

Yes. If the message won't appear, the tests run an pass

That is not a karma feature, can you explain it further? Is it IntelliJ? A button in the web page from the test framework?

Yes, it's a button in the top left corner of the karma tool window in IntelliJ to Rerun tests:
https://www.jetbrains.com/help/idea/running-unit-tests-on-karma.html#ws_js_test_rerun_failed_tests

[johnjbarton]

I think this is normal: karma launches the browser and runs the tests. Then you don't close the browser so karma gets bored and disconnects. You reload the browser and karma works fine. Rinse and repeat.

I suppose we could suppress this message in the singleRun==false case.

[aguacongas]

I experimented the same issue with Angular 8.3.20 just because in my test I wrote :

const anchorElement = fixture.nativeElement.querySelector('a');
anchorElement.dispatchEvent(new MouseEvent('click'));

And fixed it with :

const anchorElement = fixture.nativeElement.querySelector('a');
anchorElement.dispatchEvent(new Event('click'));

[rochapablo]

Even when all tested... it gets disconnected

Chrome 84.0.4147.105 (Mac OS 10.15.5): Executed 54 of 54 SUCCESS (17.303 secs / 16.793 secs)
Chrome 84.0.4147.105 (Mac OS 10.15.5) ERROR
  DisconnectedClient disconnected from CONNECTED state (transport close)
05 08 2020 14:07:29.750:INFO [Chrome 84.0.4147.105 (Mac OS 10.15.5)]: Disconnected browser returned on socket

[jwedel]

I know it has been said a lot about that Karma is used in large code bases, and this might be a problem in the browser or the application itself and it all might be true but it doesn't change the fact that there is no fix and not way to debug the issue and it is generally extremely time-consuming and annoying.

My whole team is so frustrated with this issue which happens every 1-2 test runs in multiple different applications that we've migrated our Karma/Jasmine tests to Jest which appears very stable so far.

So, good luck to the others with that issue.

[GeeWee]

I still have the error in karma 5.2. Anecdotally it also works for me when turning off source-maps, but that's not very pleasant when trying to debug something. Using Fedora 32 and Chrome

EDIT: Actually scratch that, it still happens with source maps off.

[dsypniewski]

I'm having the same problem and one thing that I noticed is that this problem is way more likely to show up after changing the code that is being tested (not the tests) and that sometimes this timeout happens while fetching big vendor package, which causes to hit the timeout and here it helps a bit but this is not the only reason why those disconnects keep happening.

The only other thing I noticed is that I get a "WebSocket is already in CLOSING or CLOSED state." message in Chrome when those disconnects happen and that's it. I think an important part here is that the tests in the browser complete just fine without any errors anywhere except this socket one when Karma tries to send a message back.

With or without those error for some reason new socket connections are being established but the weird part is that even after Karma server detecting the disconnect the WebSocket stays open and the ping/pong messages keep on going even though the server thinks it lost the connection, or maybe there is something else going on here that I'm not aware of.

[johnjbarton]

Yes those are the same symptoms we see. I believe that PR #3653 will "fix" this by removing all of the different code paths in favor of one based on pingTimeout. Then you will increase the pingTimeout or change the size of the content you send to the browser. In fact you can try that much now.

[seaders]

In my situation, it was happening after a period of testing inactivity, especially if I just left VS Code open, while doing something else, or stepping away entirely.

My "fix" is to touch the karma.conf.js file. Extremely imperfect solution, but in my case without 100s of tests, it's much faster than restarting that VS node.

[shubhamsgit]

I fixed this by changing the node version :)

[amitkrishnabits]

Op, is this because of VPN

[Mesya82]

In my situation, it was happening after a period of testing inactivity, especially if I just left VS Code open, while doing something else, or stepping away entirely.

My "fix" is to touch the karma.conf.js file. Extremely imperfect solution, but in my case without 100s of tests, it's much faster than restarting that VS node.

That actually works. Thank you.

[sn-russellm]

I also get this issue

[DmitriyStoyanov]

Faced the same issue with connections and fixed by downgrading karma package from 6.4.2 to previous major version 5.2.3.

And adding

        listenAddress: "localhost",
        hostname: "localhost",

and
"--remote-debugging-address=0.0.0.0" into ChromeHeadless flags
in karma config file