ember test --server stops building and refreshing the browser after a while

[ghedamat]

Intro

We have been noticing the same behavior across all our apps at work
and I was able to reproduce on a brand new app

The issue

After an amount of time that "seems" to be hovering around 20-30 minutes ember test --server stops sending the "start test" message over the websocket when files are changed. Even opening a new tab or a new window in a different browser does not fix the issue. All connected clients will not be notified when the code is changed.

Furthermore (although I have less proof of this) even though ember test --server seems to recompile when code is changed it looks like the assets being server are not different and changes don't make their way into the compiled assets

The only fix is to restart ember test --server

reproduction

ember new test-app
cd test-app
yarn
ember test --server
# open browser and visit test page, 
# make a change to app/app.js
# see page reload
# wait 20 min or so
# make a change to app/app.js
# page won't reload
# change is not reflected in new source code

thanks for listening, happy to provide any more context or take suggestions on how to move this forward!

---

Output from ember version --verbose && npm --version && yarn --version:

➜ ember --version
ember-cli: 3.21.0
node: 10.22.0
os: linux x64

➜ npm --version
6.14.6

➜ yarn --version
1.22.5

note that I was able to reproduce this on node 12 as well

[lukemelia]

I've been noticing this too recently, but haven't found any reproducible steps.

[rwjblue]

After an amount of time that "seems" to be hovering around 20-30 minutes ember test --server stops sending the "start test" message over the websocket when files are changed.

Does this happen when completely idle (like start ember test --server then do nothing in app or browser)?

[ghedamat]

@rwjblue I am not sure but can verify.

I also wanted to point something else out
@acorncom suggested on Discord to ensure watchman is installed.

Turns out it was not, I added it to our base setup and it seems to solve the issue for me so far.
I'm rolling out the update to the entire team and will monitor over the next few days if this helps

I can also try to repo the cold build case that you are asking about without having watchman installed.

[rwjblue]

Ya that sounds great, thank you @ghedamat!

[patocallaghan]

We've also been seeing this a large amount at Intercom. It's actually been happening for a fairly long time. We've an internal issue open to look into it since the beginning of 2020 😬 I did some digging and it seems to have something to do with some code in Testem where it will stop trying to connect to the browser (and therefore reloads) if it hits the threshold of the browser_reconnect_limit. Currently this defaults implicitly to 3 in new Ember apps and addons as we don't list it in the testem config.

To try and reproduce I followed the following process:

1. Enabled testem logging when running tests using yarn ember test -s -tp=0 --testem-debug=testem.log
2. Continuously saved the same file in the app
3. Observed a single test running in the browser
4. After a bit of time I observed that the test page stopped live reloading

See the testem logs generated below:

Testem logs showing reconnection limit being reached, i.e.

socket reconnection limit has been exceeded, not attaching new socket for browser Chrome 86.0 and id 9365

info Test'em starting.. 
info Seeking for config file... 
info Starting dev 
info Starting server 
WARN Launcher "Chrome Canary" is not recognized. 
info Waiting for tests. 
info Start triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 HjzVn5KTUdK3CqYkAAAA 
info runnerAdded Chrome 86.0 9365
info socket reconnection count: 0 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
ERR! ENOENT: no such file or directory, stat '/var/folders/mk/5t937px5627_y_j551f8gbkm0000gn/T/tests-dist-20201013-26247-1qxnmw6.ugwy/favicon.ico' 
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info New client connected: Chrome 86.0 9365 aUiXMONWYt4MCcWbAAAB 
info socket reconnection count: 0 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info New client connected: Chrome 86.0 9365 f3RM6Boj2LRuFYJLAAAC 
info socket reconnection count: 0 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 uLZUmCdxe3ETk51lAAAD 
info socket reconnection count: 0 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 uSq-g3W8TWc4yEEnAAAE 
info socket reconnection count: 0 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 5KroEHaq_OLWIF89AAAF 
info socket reconnection count: 1 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 HTNERQp35nJl-_5IAAAG 
info socket reconnection count: 2 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 sZ_1jfqwNv0rofZOAAAH 
info socket reconnection count: 3 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 GWIEZKplS5S7Txj6AAAI 
info socket reconnection count: 3 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 oB38jxbXToKRrmeYAAAJ 
info socket reconnection count: 3 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 o0bL37SM5g_4ncz6AAAK 
info socket reconnection count: 3 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 UiaLKVWwOkRh99TpAAAL 
info socket reconnection count: 3 
info tryAttach Chrome 86.0 9365
info Chrome 86.0 onStart
info Browser Chrome 86.0 finished all tests. undefined
info Chrome 86.0 onEnd
info Api: restart triggered test run. 
info Running tests... 
info Tests running. 
info New client connected: Chrome 86.0 9365 TMaFTMNuM4TmQOocAAAM 
info socket reconnection count: 4 
info socket reconnection limit has been exceeded, not attaching new socket for browser Chrome 86.0 and id 9365 
info Api: restart triggered test run. 

1. Notice for multiple reloads the reconnection count is incrementing socket reconnection count: 1 (this is a bit of extra logging I added in testem and did yarn link)
2. Once the reconnection count hits 4 it fires showing reconnection limit being reached and at this point the test page stops reloading when editing files

It's worth pointing out that I don't use Watchman (not installed on my machine). In this case anyway it seems the build happens but it's the actual reloading of the page that doesn't happen.

As to why the disconnects/reconnects are happening, I don't know. Possibly the browser_disconnect_timeout is being hit? Is the fix here to just simply bump the browser_reconnect_limit in the testem config? Although this doesn't solve the root cause.

/cc @step2yeung (who made the original change in testem/testem#1340) who may have context on why these disconnects could happen?

[timiyay]

I've been experiencing this issue chronically for the past year or so. Using ember test --server is only working for about 5-10 minutes, then it requires a restart.

[rwjblue]

Hmm, could this be due to the issue that @brendenpalmer fixed in broccolijs/broccoli-concat#156? That bug was very very difficult to diagnose, but basically caused concat to ignore changes to files over the 10000 char limit in specific contexts. This makes it appear that rebuilds are happening but the new content is never served.

Can y'all check what version of broccoli-concat is being used (and if possible force it to 4.2.5)?

[jherdman]

(responding on behalf of @ghedamat ): 4.2.4. I'll see if we can bump.

[timiyay]

We're also on 4.2.4 in all of our apps.

I'll bump to 4.2.5 and test it out next time I'm using ember test --server.

For the time being, I can workaround the issue by bumping up some testem config values to large numbers:

browser_disconnect_timeout: 3600,
browser_reconnect_limit: 1000,

[jherdman]

Initial investigation suggest that 4.2.5 does not resolve this problem.

[rwjblue]

Drats, sorry folks.

If the issues is indeed with testem configuration, we can update the default blueprint to use those settings...

[patocallaghan]

Yep I've found increasing the browser_reconnect_limit resolved the issue somewhat for us. It doesn't solve the root cause but meant livereload didn't die as quickly. #9333 (comment)

[MrChocolatine]

I have noticed this behaviour for years, perhaps since Ember.js v3.8.
/following_thread

[mwagz]

I notice this most often when I try to filter or apply a specific test module. I can't recall seeing it any other way.

[jonathanstiansen]

Anyone find an issue over the years?

[NullVoxPopuli]

we're in the process of switching the-command-that-you-build-with to webpack/vite, so unless someone wants to debug and fix this, the problem will go away with the switch to using webpack/vite directly 🎉

[jonathanstiansen]

@NullVoxPopuli Thank you! I was captivated when I saw the https://github.com/lifeart/demo-ember-vite demo. Since joining an ember company from React - and losing access to vite/esbuild/etc I've been VERY sad about build and test times 😭

[NullVoxPopuli]

@jonathanstiansen there are a couple more vite-using projects, too:

• https://github.com/NullVoxPopuli/polaris-starter
• and https://github.com/embroider-build/embroider/tree/main/tests/vite-app

there are more demo-land atm, but progress is ongoing!