gatsby with graphql won't load new data

[Necromant1k]

I have an issue - gatsby with graphql won't update data from database ( firebase ).

1. Server started ( npm run develop ) in my gatsby-node.js data fetched and dynamic pages created.
2. I'm going to add some rows into my database - added
3. Reloading page (where i should see new items ) => I see there only old items (no new appear). Same even if i will go to localhost:8000/___graphql There is only old data
4. Going to close connection and start server again
5. New data appear
   And i have to do it always to see new data.

Any solutions on this?
Thanks!

gatsby-node.js

const _ = require(`lodash`);
const Promise = require(`bluebird`);
const path = require(`path`);
const slug = require(`slug`);
const slash = require(`slash`);

exports.createPages = ({ boundActionCreators, graphql }) => {
  const { createPage } = boundActionCreators

  return new Promise((resolve, reject) => {
    resolve(
        graphql(
            `
            {
              allImports {
                edges {
                  node {
                    id,
                    opened,
                    unsubscribed,
                    uid,
                    totalCsvLength,
                    clicked,
                    complained,
                    delivered,
                    duplicate,
                    failed,
                    importStartedAt
                  }
                }
              }
            }
        `
        ).then(result => {
          if (result.errors) {
            reject(result.errors)
          }
          const importTemplate = path.resolve(`src/templates/import-detail.js`);
          result.data.allImports.edges.map(edge => {
            createPage({
              path: `/${slug(edge.node.id)}/`,
              component: slash(importTemplate),
              context: {
                id: edge.node.id,
              },
            })
          })
        })
    )
  })
};

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `example`,
  },
  plugins: [
      `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-firebase`,
      options: {
        // point to the firebase private key downloaded
        credential: require("./config/firebase.json"),

        // your firebase database root url
        databaseURL: "https://url.firebaseio.com",

        // you can have multiple "types" that point to different paths
        types: [
          {
            type: "Imports",
            path: "imports",
          }
          ]
      }
    }
  ],
}

[dardub]

While in development mode, the server will watch for changes to files and rebuild/update accordingly.

In production, this is a static site and content is only updated when you do a fresh build and deploy to your host.

There is no way for Gatsby to know that you made changes to your database. Firebase would need to trigger an event to your CI server that you need a new build.

Another approach is to not use Firebase as a source and have it update content dynamically in your app. But then you are losing some of advantages of having static compiled content.

[dardub]

Just to add, I would make sure that using Firebase is necessary. Can you not achieve the same outcome using markdown or yaml? This would make things simple and easy to maintain.

[PepperTeasdale]

What if there was an optional flag when restarting the server that allows you to add a query parameter that refetches from your data source? Perhaps, the server could accept post requests to trigger rebuilding the schema and pulling data, so data sources could use webhooks. Use case: our content editors want to preview their entries before publishing.

I would be interested in working on something like this, but could use a little guidance through the code, since I'm new to the project.

[PepperTeasdale]

Any thoughts on this proposed solution? Would gatsby be amenable to one of these or some variation thereof?

[KyleAMathews]

My preferred solution is the last idea here #2271 (comment)

We'd expose a way to run commands while gatsby develop one of which would be to refresh data from sources.

[KyleAMathews]

Perhaps, the server could accept post requests to trigger rebuilding the schema and pulling data, so data sources could use webhooks

You want to run gatsby develop on a web server somewhere that's accessible to many people? Hmmm interesting 🤔

[PepperTeasdale]

Admittedly not the greatest option, but it would be behind an oauth proxy so only users in our organization can access the site. How are other people serving up previews of their site?

[PepperTeasdale]

I like the version of using a source plugin to add new nodes. Was not aware that was a supported use case. Perhaps I could add this as an option for gatsby-source-contentful

[bjrn]

Being able to run a gatsby develop on a web server for site previews, with data coming in from gatsby-source-contentful matches my use case exactly, @PepperTeasdale — can I help out making it happen in any way?

[PepperTeasdale]

Sending you an email @bjrn

[tlvenn]

Being able to run a gatsby develop on a web server for site previews, with data coming in from gatsby-source-xxxx

This is also my use case with prismic instead of contentful. I dont really understand why we could not start gatsby develop with an additional flag that would force gatsby to refetch its data for the requested page.

Pushing that concern for this particular use case (preview mode) to each individual source plugin does not seem like the right approach.

You want to run gatsby develop on a web server somewhere that's accessible to many people? Hmmm interesting

I dont really see any other way to let authors preview their contents before it's getting published. And for many working with a headless CMS, this is really mandatory for adoption.

There is definitely a legitimate use case to run gatsby on a server in order to provide previews imho and it seems gatsby develop already provide 95% of what is needed.

It could probably be a good idea to introduce gatsby preview rather than passing an additional flag to gatsby develop.

[PepperTeasdale]

I agree with @tlvenn , a gatsby preview server sounds like the ideal solution. It could offer a basic version to allow re-bootstraping before handling requests, taking the responsibility off the plugins. Even better would be to expose an API to receive webhooks from data sources and allow source plugins to update accordingly.

I took a look at the filesystem source plugin as @KyleAMathews recommended and noticed it was using chokidar to watch filesystem changes. At first I thought I could hook into onClientEntry and use EventEmitter to trigger a re-running of sourceNodes, but then realized that browser api and node api wouldn't work together that way. Any suggestions on which parts of the API I need to be looking at to make this work?

[PepperTeasdale]

I guess ignore the latter part of the above comment, I came up with some options. @tlvenn , maybe have a look at #3054 if you'd like to join that conversation

[PepperTeasdale]

Well, my solution was working when i was running gatsby locally and using yarn link to include it in my project. Unfortunately the pages are not actually updating now that the changes were merged int. Trying to hunt down why still, but I'm thinking some data must be cached and not updating.

When the endpoint is hit, sourceNodes is called, and the CREATE_PAGES action is dispatched, the updated data is fetched from my source (contentful) and the writePages() function in src/internal-plugins/query-runner/pages-writer.js is being called. Looking at the .cache directory, the .cache/redux-state.json file has been updated, but the data .cache/json/${page-title} has not. If I manually change the latter, the page updates I've noticed. I'm going to keep digging, but if anyone has any ideas on why something would be different when running a local version of gatsby vs. the npm published version or if there have been any code changes in this area in the past couple of weeks that might affect this behavior, please let me know. Thanks!

[PepperTeasdale]

I did a git bisect and found that the sourceNodes solution stopped working with 3e2d852e64861956b9c10e46b447782aac9588c0 . @KyleAMathews might be able to give a little context, but it looks like gatsby is running source nodes, but the page query runner has kept the ID of the page in memory in seenIdsWithoutDataDependencies and therefore does not run the query for the page. In order for the existing pages to load new data, they'll need some way to bust the cache here. I would appreciate any advice on how to do this!

[KyleAMathews]

Due to the high volume of issues, we're closing out older ones without recent activity. Please open a new issue if you need help!

[osartun]

Hey guys

This conversation is already quite old, but I'm wondering what the outcome was.

We have setup right now where the editors trigger a webhook whenever they publish a page which starts a build on Netlify.
Problem is: That can take up to several minutes until the site is live. Editors waiting for several minutes before seeing the change they did significantly slows down their workflow.

So, I'm looking into solutions to that problem and also had the idea of running a development server on a web server somewhere and then make use of hot reloading. However, I need to refetch the data from our CMS (neither prismic nor contentful) ideally without stopping and starting the server.

That's how I ended up reading this thread which is probably the closest to this idea. Was the problem solved in Gatsby?
Or if not: How did you guys resolve the issue?

[rahul-nath]

I sourced my data with gatsby-graphql-source which has a refreshInterval you can input in (value is in seconds). Setting this value will pull from your datasource at that interval and run createPages. Look at the Gatsby docs on the strategies around that -- you can batch requests to make the refresh faster. Better than setting up a build endpoint imo.

[osartun]

I just randomly found my comment from 1.5 years ago (#2847 (comment)) and given the number of reactions on that comment I want to leave this here: The Gatsby Cloud service is the very idea that I'm proposing in the comment.
So, if anyone else is trying to fix this issue – save the engineering time and get yourself a subscription: https://www.gatsbyjs.com/ (I'm not getting paid to say that, unfortunately)

[AjayDhimanELS]

Hi guys, Any update on this "refreshing of graphQL content/data"?