[gatsby-source-contentful] possible bug? new contentful model field does not show up in graphql response

[yesh]

Summary

I created a new boolean field in a Contentful model and I really can't get it to show up in gatsby graphql output.

I tried the following:

• deleted .cache folder and restarted gatsby develop several times
• created new fields in the same and other models in my space
• deleted and created a new API token for the Contentful space
• modified gatsby-node.js to force a cache reset

and I don't know what else to try.
the project is a super simple portfolio for an illustrator.

I hope this thread is not OT, and that the issue is not because my incompetence.
If the latter is the case, I'm sorry for wasting anyone's time.

the issue is really strange, and I had it in other couple updates of other content type models, but I always solved it with the generation of a new API key.

thanks!

Relevant information

my server starts fine and I don't have any errors other than a deprecation warning error (node:37636) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated

Attached you can find a screenshot of the GraphiQL not autocompleting the isHightlight field, that is the one I need in this case.

I tried downloading and running the Contentful boilerplate generated with the API Key, and the response is as expected.
the field is Is highlight, in the project content type:

Welcome to the Contentful JS Boilerplate

This is a simplified example to demonstrate the usage of the Contentful CDA

Fetching and displaying Content Types ...
┌─────────┬─────────┬───────────────────────────────────────────────────────┐
│ Id      │ Title   │ Fields                                                │
├─────────┼─────────┼───────────────────────────────────────────────────────┤
│ pages   │ Pages   │ Content, Hero, Title                                  │
├─────────┼─────────┼───────────────────────────────────────────────────────┤
│ project │ Project │ Cover, Description, Images, Is highlight, Title, Year │
└─────────┴─────────┴───────────────────────────────────────────────────────┘

Environment

  Binaries:
    Node: 12.1.0 - /usr/local/bin/node
    Yarn: 1.15.2 - /usr/local/bin/yarn
    npm: 6.9.0 - /usr/local/bin/npm
  npmPackages:
    gatsby: ^2.4.3 => 2.5.5 
    gatsby-image: ^2.1.0 => 2.1.0 
    gatsby-plugin-cookiehub: ^1.0.0 => 1.0.0 
    gatsby-plugin-google-analytics: ^2.0.20 => 2.0.20 
    gatsby-plugin-react-helmet: ^3.0.12 => 3.0.12 
    gatsby-plugin-sass: ^2.0.11 => 2.0.11 
    gatsby-plugin-sharp: ^2.0.37 => 2.0.37 
    gatsby-plugin-typography: ^2.2.13 => 2.2.13 
    gatsby-source-contentful: ^2.0.58 => 2.0.60 
    gatsby-transformer-remark: ^2.3.12 => 2.3.12 
    gatsby-transformer-sharp: ^2.1.19 => 2.1.19 
  npmGlobalPackages:
    gatsby-cli: 2.5.14

File contents

gatsby-config.js (only plugins):

plugins: [
    `gatsby-transformer-remark`,
    `gatsby-plugin-react-helmet`,
    `gatsby-plugin-sharp`,
    `gatsby-transformer-sharp`,
    {
      resolve: `gatsby-plugin-sass`,
      options: {
        data: `@import 'src/styles/variables';`,
        includePaths: []
      }
    },
    {
      resolve: `gatsby-source-contentful`,
      options: {
        spaceId: `snbyslhhwz06`,
        accessToken: process.env.CONTENTFUL_ACCESS_TOKEN,
        downloadLocal: true
      },
    },
    {
      resolve: `gatsby-plugin-google-analytics`,
      options: {
        trackingId: `redacted`,
        head: true,
        anonymize: true,
        respectDNT: true,
        cookieDomain: `redacted`
      }
    },
    {
      resolve: `gatsby-plugin-cookiehub`,
      options: {
        cookihubId: `redacted`,
        trackingId: `redacted`,
        head: false,
        anonymize: true
      },
    }
  ]

gatsby-node.js:

const path = require('path')

exports.createPages = ({graphql, actions}) => {
  const {createPage} = actions
  return new Promise((resolve, reject) => {
    const projectTemplate = path.resolve('src/templates/project.js')
    resolve(
      graphql(`
        {
          allContentfulProject (limit:100) {
            edges {
              node {
                id
                title
              }
            }
          }
        }
      `)
      .then((result) => {
        if (result.errors) {
          reject(result.errors)
        }
        result.data.allContentfulProject.edges.forEach(edge => {
          const title = edge.node.title,
                slug = title.normalize('NFD').replace(/[\u0300-\u036f]/g, '').replace(/\W/g, '-').replace(' ', '-').toLowerCase()
                
          createPage ({
            path: slug,
            component: projectTemplate,
            context: {
              slug: slug
            }
          })
        });
        return
      })
    )
  })
}

[LuudJanssen]

I'm using gatsby-source-contentful myself, so here's just a couple of sanity checks that have helped me in the past:

• Have you checked if the Field ID of the Is highlight is isHighlight? Contentful doesn't allow you to update the Field ID once you've created a field, so if you change the name the Field ID stays the same.
• Have you refreshed the GraphiQL interface as well after running gatsby develop again?
• Have you saved the changes to your content model in Contentful?
• Do you have an entry where the Is highlight field has a value and is it published(not a draft)?

Very simple checks, but just to make sure it's none of these.

[yesh]

• Have you checked if the Field ID of the Is highlight is isHighlight? Contentful doesn't allow you to update the Field ID once you've created a field, so if you change the name the Field ID stays the same.

sure

• Have you refreshed the GraphiQL interface as well after running gatsby develop again?

thousands times, removing cache and restarting the local server too.

• Have you saved the changes to your content model in Contentful?

absolutely

• Do you have an entry where the Is highlight field has a value and is it published(not a draft)?

yep, but it should work anyway returning a null if there is not value.

the thing is that, as I experienced it a couple of times before (e.g.: with the cover field), it's like something gets stuck randomly.
I "solved" it just restarting the local server, deleting .cache folder dozen times and ultimately creating a new API token to force a "refresh".
and this without changing anything in the contentful models.

[yesh]

Update:

I started the server 15 minutes ago and it worked, restarted it just to make sure I was not dreaming and the issue reappeared. this without making any changes to code or contentful. it's getting ridiculous :D

I just recoded and uploaded a video now to show the issue in action.

https://www.youtube.com/watch?v=I4hvsIaiYR8

[LuudJanssen]

Sure seems like a bug. I dived into the code and it seems they're using a sync token to get delta updates of the contentful data. However, I can't find where the contentful data is cached (whether that's actually the .cache folder or a different one)? If it's a different folder, that would explain why your problems would persist when deleting the .cache folder.

If it is stored in the .cache folder, the only thing I can come up with that could explain this problem is Contentful's own caching mechanism. Maybe Contentful is caching the calls the plugin makes to get the data? However that should be something specific to the plugin, otherwise your contentful boilerplate would also show cached data.

I'm handing it over to the core team because this is where I'm stuck and probably would require knowledge of the plugin's inner workings.

[yesh]

I'm handing it over to the core team because this is where I'm stuck and probably would require knowledge of the plugin's inner workings.

thanks anyway @LuudJanssen.
I think this is pretty much cache related. I just found plenty of closed issues like this one without solution :(

like this one: #3495

[wardpeet]

@pieh I believe you have more insights in this one.

[pieh]

Hey, do you have any values set for that field in any of your entries? Right now we infer schema from data, so if no entry has any values, we can't infer type and that's why field is missing.

There is open PR that will fix that ( #12816 ), but it didn't land yet. But in meantime please make sure at least one entry set isHighlight (in this example) to either true or false

[pieh]

I've missed some comments before my earlier reply:

yep, but it should work anyway returning a null if there is not value.

as mentioned in my comment - right now we rely on inferring so if there are only null values, we can't infer and can't create field on graphql because we can't know the type (without mentioned PR)

the thing is that, as I experienced it a couple of times before (e.g.: with the cover field), it's like something gets stuck randomly.
I "solved" it just restarting the local server, deleting .cache folder dozen times and ultimately creating a new API token to force a "refresh".

It would be great to be able to reproduce it if this happens when you have actual content in those fields, not quite sure how to approach it yet

[pieh]

Looking at the code - one thing comes to mind:
we store sync token for next fetch before we actually process updates

gatsby/packages/gatsby-source-contentful/src/gatsby-node.js

Lines 138 to 144 in ec038e9

	// Update syncToken
	const nextSyncToken = currentSyncData.nextSyncToken
	// Store our sync state for the next sync.
	const newState = {}
	newState[createSyncToken()] = nextSyncToken
	setPluginStatus(newState)

- this means that if something fails to process updates we will miss those in next sync which can cause issues, but it's hard to know what fails - do you get any errors in terminal when this is happening?

[yesh]

Right now we infer schema from data, so if no entry has any values, we can't infer type and that's why field is missing.

thank you @pieh, after this note of yours I finally got it to work after the fifth .cache delete e restart, and I can confirm that the issue is in fact the "no entry with value" case.

I didn't solve it earlier because, stressed out by the issue, after putting a value in an entry I tried only one time to restart it, and then erratically tried with different approaches.

Looking at the code - one thing comes to mind:
we store sync token for next fetch before we actually process updates
...
this means that if something fails to process updates we will miss those in next sync which can cause issues, but it's hard to know what fails - do you get any errors in terminal when this is happening?

so this could be the cause of the need of severals "resets" of cache and server to get it to work after an error?

this is everything I get after starting the server:

info ℹ ｢wdm｣: Compiling...
error GraphQL Error Encountered 1 error(s):
- Unknown field 'isHightlight' on type 'ContentfulProject!'.

      file: /Users/yesh/work/francescariz/src/pages/index.js


error (node:44724) DeprecationWarning: Passing lineNumber and colNumber is deprecated to @babel/code-frame. Please use `codeFrameColumns`.
 ERROR  Failed to compile with 1 errors          3:56:59 PM

 error  in ./src/pages/index.js

Module Error (from ./node_modules/gatsby/node_modules/eslint-loader/index.js):

/Users/yesh/work/francescariz/src/pages/index.js
  66:11  error  Cannot query field "isHightlight" on type "ContentfulProject"  graphql/template-strings

✖ 1 problem (1 error, 0 warnings)


 @ ./.cache/sync-requires.js 17:54-114
 @ ./.cache/app.js
 @ multi event-source-polyfill ./node_modules/webpack-hot-middleware/client.js?path=/__webpack_hmr&reload=true&overlay=false ./.cache/app

error ✖ ｢wdm｣:
ERROR in ./src/pages/index.js
Module Error (from ./node_modules/gatsby/node_modules/eslint-loader/index.js):

/Users/yesh/work/francescariz/src/pages/index.js
  66:11  error  Cannot query field "isHightlight" on type "ContentfulProject"  graphql/template-strings

✖ 1 problem (1 error, 0 warnings)

 @ ./.cache/sync-requires.js 17:54-114
 @ ./.cache/app.js
 @ multi event-source-polyfill ./node_modules/webpack-hot-middleware/client.js?path=/__webpack_hmr&reload=true&overlay=false ./.cache/app
info ℹ ｢wdm｣: Failed to compile.

thank you again @pieh and @LuudJanssen for your help

PS: would it be good to expand the gatsby-source-contentful documentation with a note about this issue?

[pieh]

Those errors are result of missing field and not cause for it, so those won't help in tracing down potential cache issue (if there is cache issue here - hard to determine that without concrete reproduction steps that fails when at least one entry in contentful has data for that missing field).

PS: would it be good to expand the gatsby-source-contentful documentation with a note about this issue?

This is general gatsby issue and not just gatsby-source-contentful. We very recently added ability in gatsby core to define fields so we don't have to rely on data, but plugin ecosystem wasn't yet adjusted to use this new API. Contentful is actually one of the first official plugins that will use that APIs (once the PR I linked above will be finished). There are already some that use those APIs - like plugins for Sanity or DatoCMS

[gatsbot]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contributefor more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[gatsbot]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[samuelgoldenbaum]

Can confirm the following which may help in reproducing:

1. I have a nullable DateTime content field called withdrawn on a content type
2. If I have 10 records and only 1 of them has the withdrawn field filled, if I then unpublish that particular record - leaving 9 records with a null withdrawn value, then the problem is raised and the query cannot include the missing withdrawn field

[vasupol11]

I found that it looks at the entries rather than the model to figure out the schema. So after adding a new field to the model just make sure to make one of the entries have that field configured. For example, I had to add an image to my new media field to make gatsby-contentful see it.

[ghost]

found a solution that worked for me. in package json add the following script and run npm run clean. cleans the cache. this is a cache issue with gatsby.

{ "scripts": { "clean": "gatsby clean" } }

[gamsim]

The only thing that worked for me was to update the content model and update / publish new content from that updated model.