Gatsby doesn't always refresh changes

[tu4mo]

Description

Gatsby sometimes fails to reload changed components.

Issue can be reproduced after version v2.24.9, so something changed between 2.24.9-2.24.10.

Steps to reproduce

https://github.com/tu4mo/gatsby-reload-issue

git clone git@github.com:tu4mo/gatsby-reload-issue.git
cd gatsby-reload-issue
npm install
npm start

Run (multiple times if needed) in a another terminal:

node update-fakelib.js

Expected result

The scripts rewrites 10 components with current timestamp. All changed components are hot reloaded.

Actual result

Some, all or none of the components are updated, inconsistently.

Environment

  System:
    OS: macOS 11.0.1
    CPU: (8) x64 Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.15.1 - ~/.nvm/versions/node/v14.15.1/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 6.14.8 - ~/.nvm/versions/node/v14.15.1/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 87.0.4280.88
    Edge: 87.0.664.57
    Firefox: 83.0
    Safari: 14.0.1

[vladar]

Oh, I see you've already followed my advice: #26580 (comment) 😄

We need some way to reproduce it to move forward. Does it happen for you on any project, including our blog starter? If not, please provide a minimal reproduction and we'll be able to take another look.

[tu4mo]

Yes, I can reproduce it with blog starter. Typing something in index.js, saving quickly twice (cmd-S), making a change, saving quickly twice; doing this a few times and you'll see that the change isn't shown on the page.

[tu4mo]

[tu4mo]

I also reversed blog starter to a commit that had a version later than v2.24.0 (v2.23.12) and I can't reproduce the issue there. Clearly something changed in the refresh mechanism after v2.24.0 and made it unreliable.

@vladar Is there anything else I can do to help with this issue getting fixed? Or am I forever stuck with v2.24.0... 😞

[vladar]

I tried to reproduce it with the latest gatsby version and also on 2.24.0 but I don't see stale results. Which makes me think that it may have something to do with your environment.

Do you have Dropbox or other similar tools installed in the same folder where your gatsby site is located? Maybe some file watchers in IDE or other tools?

[vladar]

Also, can you bisect a specific version where this problem was introduced for you? The 2.23.12 - 2.24.0 range has too many changes. We need to narrow it down.

[tu4mo]

I don't have Dropbox or any other file sharing apps using code files. In my project I have watchers that built the library (Rollup and TypeScript in watch mode).

Sorry for not being clear about the version. 2.24.9 works, 2.24.10 does not.

[vladar]

Then it sounds very similar to #26192 Have you tried upgrading to 2.24.57 as suggested here: #26192 (comment) ?

[tu4mo]

Tried 2.24.57, same thing, does not reload.

[tu4mo]

It feels like there's something little different in 2.24.57. After I edited a component in my library, I did get it reload by editing a Gatsby page where the component is imported. Sometimes it works, sometimes it doesn't, sometimes the whole reloading seems to freeze, and I have to restart Gatsby.

[vladar]

After I edited a component in my library, I did get it reload by editing a Gatsby page where the component is imported. Sometimes it works, sometimes it doesn't, sometimes the whole reloading seems to freeze, and I have to restart Gatsby.

But can you reproduce the problem on the blog starter with gatsby@2.24.57? If not - then it means we are still stuck without any reproduction that we could use to troubleshoot your specific issue.

[tu4mo]

Yes, I can reproduce it everytime typing rapidly aaa, cmd-S, cmd-S, bbb, cmd-S, cmd-S, ccc, cmd-S, cmd-S, etc.

[tu4mo]

https://github.com/tu4mo/gatsby-reload-issue

git clone git@github.com:tu4mo/gatsby-reload-issue.git
cd gatsby-reload-issue
npm install
npm start

In a another terminal:

node update-fakelib.js

With v2.24.57 I rarely get to last update. Running it a few times and the page stops responding.

With v2.24.0 it works.

You might need to increase the amount of components "created" and decrease the sleep.

[vladar]

Thank you so much for reproduction! 💜

I definitely see how develop is getting stuck eventually. Also in the given version range - #25815 looks like a primary suspect.

Investigating this.

[vladar]

So one potential problem is here:

gatsby/packages/gatsby/src/state-machines/develop/index.ts

Lines 189 to 195 in 441263e

	recompiling: {
	invoke: {
	src: `recompile`,
	onDone: {
	actions: `markSourceFilesClean`,
	target: `waiting`,
	},

We call markSourceFilesClean upon finishing recompiling JS. But all changes that occur during recompile are discarded. That doesn't explain why it doesn't recover on subsequent file changes when in waiting state.

One possible hint - somehow webpack stops calling invalid hook (even though chokidar actually emits change events for those):

success Building development bundle - 5.377s
[chokidar_change] fakelib\Component.js
[webpack.compiler.hooks.invalid] fakelib\Component.js
success onPreExtractQueries - 0.004s
success extract queries from components - 0.062s
success write out requires - 0.001s
[chokidar_change] fakelib\Component.js
success Re-building development bundle - 0.204s
[webpack.compiler.hooks.invalid] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js
[chokidar_change] fakelib\Component.js

So the next step is to dive deep into suspend/resume of the webpack. Just leaving some notes here to keep the context around.

[vladar]

OK. I think I've found the issue. This PR should fix it (hopefully): #28237

[vladar]

Published in gatsby@2.27.2. Can you please check if it fixes the issue for you?

[tu4mo]

Tested, and it's better as in it doesn't crash anymore. But it still fails refreshing most of the time :(

Tried editing a Badge component in my library 4 times. Here is the browser log.

1. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 10138ms

Component didn't update.

2. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 11960ms
[HMR] Checking for updates on the server...
[HMR] Updated modules:
[HMR]  - ../lib/dist/lib/components/Badge/Badge.js
[HMR] App is up to date.

Component updated.

3. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 11997ms
[HMR] bundle rebuilding
[HMR] bundle rebuilt in 419ms

Component didn't update.

At that time I also noticed this in terminal, not sure if related:

warning Warning: Event "xstate.after(200)#waitingMachine.aggregatingFileChanges" was sent to stopped service "waitingMachine". This service has already reached its final state, and will not transition.
Event: {"type":"xstate.after(200)#waitingMachine.aggregatingFileChanges"}

4. edit:

[HMR] bundle rebuilding
[HMR] bundle rebuilt in 532ms

Component didn't update.

Gave up after that.

[tu4mo]

I also see this in the terminal 4 times:

info changed file at /Users/tu4mo/Code/styleguide/lib/components/Badge/Badge.tsx

So Gatsby seems to pick up the change, but the rebuild times are much shorter.

[tu4mo]

I also updated Gatsby in https://github.com/tu4mo/gatsby-reload-issue and I can still reproduce this emulating multiple component updates.

[tu4mo]

@vladar Maybe you could re-open this issue?

[tu4mo]

Sorry but it didn't. Still reproducible with gatsby-reload-issue and in my real project.

[vladar]

There was another issue with webpack v5 which is hopefully fixed by #30193. Can you try again with gatsby@3.1.0-alpha-hmr-fix.11+ef7378ea10?

[tu4mo]

With 3.1.0-alpha-hmr-fix.11+ef7378ea10:

npm start
node update-fakelib.js

Components do not update, and this appears in console:

After hard-reload I see the updated components.

Second attempt with node update-fakelib, components update 🎉
Third try with node update-fakelib, components don't update 💔

[vladar]

Yeah, there were actually several underlying issues. And one was a race condition that was not that easy to repro. Can you please try again with gatsby@3.1.0-alpha-hmr-fix.13+92e67e9f09

I've tried an alternative approach there: synchronizing with webpack events properly and not relying on webpack working correctly with any sequence of resume/suspend calls at random moments.

[tu4mo]

I couldn't reproduce anymore with gatsby-repro-issue. I'll have to figure out if I can find another way.

On a larger scale, in my real app, I'm still having the old behavior though 😕. Here's a log of making changes in a Gatsby page. It reloaded changes a couple of times, but most of the times it didn't (those <1000ms rebuilds).

[vladar]

I suspected that there are several distinct issues there. Can you try again with gatsby@3.1.0-alpha-hmr-fix.16+831f9a9d64. If it doesn't help we will need a new repro (as the existing repro likely hits a different issue than your main project).

[tu4mo]

Definitely not reliable as v2.24.9 but this version seems to work almost everytime on my real project. I did get the same error as in #27609 (comment) after a few edits thought.

[vladar]

Opened an issue for the first bug in the webpack repo: webpack/webpack#12882

Still need to find a good repro for the second bug (edit: second issue: webpack/webpack#12898).

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[tu4mo]

Not stale.

[pirtlj]

Yeah this is still an issue for me

[dustinlacewell]

Does this issue have anything to do with the behavior of Gatsby not knowing about any of the types it should infer from gatsby-node queries until I hit the __refresh endpoint?

[tu4mo]

@vladar webpack v5.37.1 was just released with watching fixes.

[vladar]

Have you tried it? Maybe remove yarn.lock and try on your project? Gatsby has this dependency definition for webpack "webpack": "^5.35.0", so it shouldn't block you from upgrading?

[tu4mo]

Yes, I tried it. I have another issue with Gatsby v3 so I haven't yet been able to completely switch to v3, but regarding this issue things are looking good 🎉

[pbxr250]

In my case the naming of components have caused the issue

[tu4mo]

I suggest opening a new issue. This particular issue that I had has been resolved in webpack. Closing this.

[tu4mo]

Extra special thanks to @vladar for sticking with me through this!

[runemadsen]

I just want to echo @pbxr250 comment: I was running into a bunch of issues related to hot module reloading, and it was only in a single component. Basically, my first change to either the CSS or JS would show in the browser, but all following edits would not. There were no warnings or errors in the console.

I finally figured out it was because of a camel-case problem. My component was named WhatWeAreBuilding.js and it was imported as WhatWeArebuilding.js (with a lower-case b). In my humble opinion, this should have thrown an import error, as that file doesn't exist. Instead, it seems to work, but only for the first hot module reload.

[USAIPOST]

@runemadsen camel-case solution worked for me. When I switched branches, one of my file names had more than one uppercase characters in a row. I had to fix the file name.

[jayantamadhav]

In my case the naming of components have caused the issue

Thanks, this was the issue in my case. I named the file with capital first and then was importing that component using all lowercase.

Suggestion: Maybe make import checks case-sensitive?