refactor(gatsby): enable running of static/page queries separately

[Moocar]

Description

As part of my work to remove the global data.json, I need to run the build-javascript step after static queries have been run, but before page queries. This is because each page result file also includes the webpack build's compilation hash, so that the running app can be invalidated if a rebuild occurs while a user is navigating the page (more info in #11982).

While I was doing this refactor, I also took the opportunity to remove a bunch of global state/event emitters. The biggest is that the query queue is now instantiable. So now we can create a queue that processes the static queries, and then create a new one that processes the page queries. Then, we can start a fresh queue processor once gatsby develop has started. Hopefully it makes that part of the code easier to reason about.

I also took a stab at making the calculation of dirty queryIds a bit easier to read. It's not great, but hopefully these changes make it a little easier to understand.

page-query-runner has become index. Apologies for the lack of a good diff, but a lot of that file has changed. I have a follow up branch that moves src/internal-plugins/query-runner/ to src/query/

Related Issues

• feat(gatsby): Move page component state & side effect handling to xstate #11897

[Moocar]

@KyleAMathews this fixes the

  // HACKY!!! TODO: REMOVE IN NEXT REFACTOR
  activity.start()
  emitter.emit(`START_QUERY_QUEUE`)
  await queryRunner.processQueries(
  // END HACKY

https://github.com/gatsbyjs/gatsby/pull/12891/files#diff-8c8b888e741f1c31e97d4bf05894f50dL476

[pieh]

I think we want to keep those - those add extra overhead which is only needed in watch mode - but for production builds we don't really benefit from those

[Moocar]

Oh wow, that was accidental. I didn't mean to delete that part. Great catch

[Moocar]

Resolved in 60d1241. In my future branch, the websocket stuff is passed in by gatsby develop, so cleans up this file further.

[pieh]

Can you expand, why splitting query running is needed? Not including page queries result paths in file that gets webpacked seems like would achieve same result without touching so much code? What am I missing?

[Moocar]

Each page query result will now be in its own page-data.json file. Which looks like:

{
  "componentChunkName": "component---src-pages-index-js",
  "path": "/",
  "compilationHash": "6caec38909d092c4e3d3",
  "data": {
    "allFoo": {
      ...
    }
  },
  "pageContext": {
    "isCreatedByStatefulCreatePages": true
  }
}

The compilation hash references the latest webpack compilation. So each page-data.json depends on the build-javascript stage. So they need to be run in separate phases. I'm still working on the code that ends up using this, but you can see a preview at https://github.com/Moocar/gatsby/blob/per-page-manifest/packages/gatsby/src/commands/build.js#L65.

[Moocar]

@pieh I'm trying to break up my huge data.json PR into multiple small ones to assist with reviewing. But recognize that it makes the context harder to understand. I think it's worth it, but let me know if you'd like the full PR instead (which isn't quite finished yet)

[KyleAMathews]

Couldn't we just append the compilationHash to the page-data.json file? I think we'll need to add that capability anyways to support my image prefetching RFC as we won't discover which images need prefetching until we're rendering HTML.

[KyleAMathews]

It's more disk I/O but should be relatively cheap. Tried googling how to append to a JSON file (which would be cheapest) but it doesn't look straightforward. We should be fine though since files will almost always be < 100kb.

[KyleAMathews]

curious about the terminology — a daemon is normally a separate process right?

[Moocar]

Yeah, I'm not sold on daemon as terminology either. I'm trying to communicated that it's a long running process that won't end. Other ideas? startListener, startProcess?

[KyleAMathews]

@pieh and I and some others have talked about modelling more of Gatsby's internals after the Actor Model. It's a larger discussion to conclude if we want to do that but "startActor" would be a natural name then.

[Moocar]

If it is actually an actor, then for sure :). But until then, we should probably stick to something generic so we don't confuse people (and yes, daemon is also a bit confusing).

[KyleAMathews]

If it believes it's an actor it is! Haha.

Service is a nice generic name :-)

[Moocar]

Yep, I like service. Will change

[Moocar]

Couldn't we just append the compilationHash to the page-data.json file? I think we'll need to add that capability anyways to support my image prefetching RFC as we won't discover which images need prefetching until we're rendering HTML.

Just caught up with @KyleAMathews and we chatted about this. While we could figure out a way to load/rewrite the page-data.json after the build-javascript phase, there's technically no reason we need to finish running page queries before hand. Given that, it's simpler to write the page-data.json once after build javascript has finished.

In fact, it enables interesting possibilities like running build-javascript on another process :D

[Moocar]

I've made a few more changes downstream of this branch. I'm going to close this and reopen (or recreate) once those changes are settled.