Add endpoint to refresh external data #3054

PepperTeasdale · 2017-11-28T16:18:41Z

Fixes #2847 . The external source data can be refreshed by making a POST request to
the __refresh endpoint. If the env var GATSBY_REFRESH_TOKEN is present
there needs to be a matching Authorization header. This post request
is wrapped for convenience with the command gatsby refresh which uses
localhost:8000 by default, but allows for a host and port to be
specified in the same way as gatsby develop.

gatsbybot · 2017-11-28T16:24:38Z

Deploy preview ready!

Built with commit 64c6d2a

https://deploy-preview-3054--gatsbygram.netlify.com

gatsbybot · 2017-11-28T16:26:57Z

Deploy preview ready!

Built with commit 64c6d2a

https://deploy-preview-3054--using-drupal.netlify.com

bjrn · 2017-11-28T22:14:00Z

This would indeed be very useful — rather than having every plugin coming up with a way to refresh data, having a gatsby preview would enable an easy way to set up a preview server for a project, and have it update on changes, regardless where the data comes from; contentful, prismic etc.

PepperTeasdale · 2017-11-30T15:29:59Z

@KyleAMathews curious if this is something you would consider supporting before getting deeper into development. If not, could we expose some way to add a community-supported preview server extension (or if this is already supported, please let us know how)?

KyleAMathews · 2017-11-30T17:05:17Z

rather than having every plugin coming up with a way to refresh data

That was never the idea — all source plugins should be idempotent i.e. Gatsby can call them whenever it wants. That's why I've been suggesting adding an extension to gatsby develop that lets developers trigger refetching data from sources.

KyleAMathews · 2017-11-30T17:05:40Z

Exactly like this PR is doing :-)

KyleAMathews · 2017-11-30T17:06:18Z

@PepperTeasdale I think a built-in preview "server" makes a ton of sense

PepperTeasdale · 2017-11-30T18:53:32Z

Awesome! I'll start by fleshing out this webhook to refresh the dev server, since its something we'd probably like in both dev and preview. Then, preview feature would be more focused on a similar webpack config, but without HMR and more efficient source maps, etc.

PepperTeasdale · 2017-12-04T18:23:46Z

@KyleAMathews 👀 please :) No longer a WIP. I would like to add some tests, but wasn't seeing where this kind of stuff was being tested in the project. If you have any ideas on that, please let me know.

The external source data can be refreshed by making a POST request to the `__refresh` endpoint. If the env var GATSBY_REFRESH_TOKEN is present there needs to be a matching `Authorization` header. This post request is wrapped for convenience with the command `gatsby refresh` which uses localhost:8000 by default, but allows for a host and port to be specified in the same way as `gatsby develop`.

bjrn

Great work! Covers all things we've been looking for in a preview server setup, right down to elegant handling of the refresh_token. And with a lot less code changes than I had imagined!

@KyleAMathews is there anything preventing this from getting merged? I guess this feature falls under advanced usage, but is more documentation needed? An example setup?

bjrn · 2017-12-07T14:42:17Z

packages/gatsby/src/commands/develop.js

+   * If no GATSBY_REFRESH_TOKEN env var is available, then no Authorization header is required
+  **/
+  app.post(`/__refresh`, (req, res) => {
+    if (req.headers.authorization === process.env.GATSBY_REFRESH_TOKEN) {


nitpick or misunderstanding perhaps, but if process.env.GATSBY_REFRESH_TOKEN is undefined, shouldn't refresh happen regardless of what is passed in res.headers.authorization?

Thanks for having a look! You point about the authorization headers makes sense. Also intend on adding appropriate documentation. Just wanted to handle any requested changes to the implementation details before writing it out (eg if the refresh token is defined as a command line argument when starting the server, rather than an env var).

I think I'd personally vote for it to be an env var

PepperTeasdale · 2017-12-11T15:06:45Z

I was thinking about it this weekend, and it might make sense to open this up as a hook for plugins, something like onDataRefresh. The default behavior might be to just refetch and build everything, but allow the user to define plugins that can be smarter about how to handle the POST request, based on the data received.

Nothing here would prevent future development of something like that.

KyleAMathews · 2017-12-12T17:42:38Z

packages/gatsby-cli/src/create-cli.js

@@ -105,6 +105,25 @@ function buildLocalCommands(cli, isLocalSite) {
    handler: getCommandHandler(`develop`),
  })

+  cli.command({


Are you assuming that you'd have gatsby develop running and then run this in another terminal tab? If so, this wouldn't work as Gatsby keeps the node data in memory — we need to add the ability to pass commands to the running gatsby develop process e.g. press "r" and a refresh would be triggered.

Thanks for looking! This does work (in my local tests, at least), as it makes a post request to the port the server is running on, hitting an endpoint that calls sourceNodes().

If I understand correctly, you envision that the process running gatsby develop is also listening to stdin and updating based on input there, (eg: pressing "r" in the terminal window). I'm not familiar with any programs that do that, but I can totally give that a shot.

Even without the refresh command, would you be open to the dev server accepting a webhook to reload the data? That is my use case and what some others commenting on the related issues are looking for.

ah, pressing r as in reload (kind of like how Jest CLI works?) could be nice for local development, but I agree with @PepperTeasdale that the main use case would be to use the webhook … since it's just a POST, building a trigger that fits the requirements on top of it would be trivial.

PepperTeasdale · 2017-12-19T14:55:18Z

@KyleAMathews Just bumping to see if you would be interested in supporting the webhook portion of this PR. And if you would, what changes would be required for a merge? Thanks!

KyleAMathews · 2017-12-19T19:16:22Z

@PepperTeasdale yeah! Let's ship it first w/o CLI command so we can test it w/o confusing people initially for what it's for. That gives y'all more flexibility for trying stuff early on. Later we can add the CLI command & documentation, etc.

KyleAMathews · 2017-12-19T19:16:30Z

Dark deploy :-)

PepperTeasdale · 2017-12-20T22:57:24Z

Thanks @KyleAMathews ! Closing this and putting in the slimmed down PR at #3293

wysher · 2018-03-11T22:02:55Z

@PepperTeasdale, @KyleAMathews
What about mentioned docs? :) Or maybe some simple example how to use it.

PepperTeasdale · 2018-03-11T22:20:22Z

@wysher I need to make a PR to remove this code. It broke after a big fix went in. We ended up abandoning this path in our efforts to make a preview server.

camsloanftc · 2018-03-20T19:24:29Z

@PepperTeasdale - when you say you're going to submit a PR to remove this code, which code are you referring to? The code from #3293? I am just curious because we are about to set up a preview server and we were planning to use the refresh endpoint you added in that PR. Could you let me know if that will be removed soon then? It will change our course of action.

PepperTeasdale · 2018-03-21T16:39:22Z

@camsloanftc I do think that #3293 should be removed
@KyleAMathews merged that PR to allow us to test that idea, but it didn't pan out. See my comment on the original issue for more context. I'm sorry I didn't think to do this sooner. You could certainly investigate this route further for your preview server, but I don't think it would work out the box (at least last time I checked).

PepperTeasdale · 2018-03-21T16:46:40Z

Although @Jrousell said they got it working last week, so perhaps I am wrong about this.

camsloanftc · 2018-03-23T03:10:40Z

I had it working before as well, with the exception of some Contentful data that seemed to be stuck in cache when attempting to delete content. I will try to set it up again to see what the status is.

camsloanftc · 2018-04-11T21:43:01Z

I was just testing out this feature again and had it working fairly well. I made a video of the process. I will summarize it here though:

I added the environment variable: ENABLE_GATSBY_REFRESH_ENDPOINT=true to my .env file

Then I setup a webhook in Contentful to access my local machine. I used ngrok to make a secure tunnel to do so. This is just for demonstration, usually this would point to a real server.

Once I run gatsby develop on my machine, in Contentful I was able to update content, titles, etc.
I could rearrange a list of linked fields, or delete linked items.

Where I ran into trouble was archiving or deleting items. It does not seem to bust through what is stored in the .cache folder. I think this has been referenced in a separate issue, (#3495 maybe).

If there is interest in seeing a video of this in action, I have it posted here.

It seems to me like we may not need to remove this code, but figure out the issues with the cache.

KyleAMathews · 2018-04-14T16:51:17Z

@camsloanftc very cool video! Looks like this TODO just needs done https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-contentful/src/gatsby-node.js#L73-L81

Khaledgarbaya · 2018-04-16T09:19:21Z

👋 @camsloanftc @KyleAMathews that can be easily done by https://www.contentful.com/developers/docs/references/content-delivery-api/#/reference/search-parameters/links-to-entry
You can query contentful for all entries that reference the now deleted entry/asset

PepperTeasdale mentioned this pull request Nov 28, 2017

gatsby with graphql won't load new data #2847

Closed

PepperTeasdale changed the title ~~WIP add endpoint to refresh external data~~ PROPOSAL add endpoint to refresh external data Nov 29, 2017

PepperTeasdale changed the title ~~PROPOSAL add endpoint to refresh external data~~ [PROPOSAL] add endpoint to refresh external data Nov 29, 2017

ivorpad mentioned this pull request Nov 29, 2017

Dynamically fetch data from a blog with GraphQL #3068

Closed

PepperTeasdale force-pushed the refresh-data-sources branch 2 times, most recently from c4685aa to 213981d Compare December 4, 2017 18:21

PepperTeasdale changed the title ~~[PROPOSAL] add endpoint to refresh external data~~ Add endpoint to refresh external data Dec 4, 2017

PepperTeasdale force-pushed the refresh-data-sources branch from 213981d to 2841ada Compare December 4, 2017 18:25

PepperTeasdale force-pushed the refresh-data-sources branch from 2841ada to 64c6d2a Compare December 4, 2017 18:45

bjrn reviewed Dec 7, 2017

View reviewed changes

KyleAMathews reviewed Dec 12, 2017

View reviewed changes

PepperTeasdale mentioned this pull request Dec 20, 2017

Add endpoint for refresh #3293

Merged

PepperTeasdale closed this Dec 20, 2017

KyleAMathews mentioned this pull request Apr 16, 2018

[gatsby-source-contentful] Gatsby unaware of change in Contentful unless delete cache folder #3495

Closed

brendanmc6 mentioned this pull request Oct 15, 2018

Fetch new content in development jbolda/gatsby-source-airtable#21

Closed

m-allanson mentioned this pull request Dec 6, 2018

Document ENABLE_GATSBY_REFRESH_ENDPOINT #10328

Closed

This was referenced Sep 17, 2021

[Snyk] Fix for 2 vulnerabilities Xtuden-com/gatsby#1124

Open

[Snyk] Fix for 2 vulnerabilities nidhi42/gatsby#313

Open

[Snyk] Security upgrade prismjs from 1.24.1 to 1.25.0 MaxMood96/gatsby#173

Open

snyk-bot mentioned this pull request Feb 19, 2022

[Snyk] Fix for 15 vulnerabilities 0xSebin/gatsby#10

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add endpoint to refresh external data #3054

Add endpoint to refresh external data #3054

PepperTeasdale commented Nov 28, 2017 •

edited

gatsbybot commented Nov 28, 2017 •

edited

gatsbybot commented Nov 28, 2017 •

edited

bjrn commented Nov 28, 2017

PepperTeasdale commented Nov 30, 2017

KyleAMathews commented Nov 30, 2017

KyleAMathews commented Nov 30, 2017

KyleAMathews commented Nov 30, 2017

PepperTeasdale commented Nov 30, 2017

PepperTeasdale commented Dec 4, 2017 •

edited

bjrn left a comment

bjrn Dec 7, 2017

PepperTeasdale Dec 7, 2017

bjrn Dec 7, 2017

PepperTeasdale commented Dec 11, 2017

KyleAMathews Dec 12, 2017

PepperTeasdale Dec 12, 2017 •

edited

bjrn Dec 13, 2017

PepperTeasdale commented Dec 19, 2017

KyleAMathews commented Dec 19, 2017

KyleAMathews commented Dec 19, 2017

PepperTeasdale commented Dec 20, 2017 •

edited

wysher commented Mar 11, 2018

PepperTeasdale commented Mar 11, 2018

camsloanftc commented Mar 20, 2018

PepperTeasdale commented Mar 21, 2018

PepperTeasdale commented Mar 21, 2018

camsloanftc commented Mar 23, 2018

camsloanftc commented Apr 11, 2018

KyleAMathews commented Apr 14, 2018

Khaledgarbaya commented Apr 16, 2018

Add endpoint to refresh external data #3054

Add endpoint to refresh external data #3054

Conversation

PepperTeasdale commented Nov 28, 2017 • edited

gatsbybot commented Nov 28, 2017 • edited

gatsbybot commented Nov 28, 2017 • edited

bjrn commented Nov 28, 2017

PepperTeasdale commented Nov 30, 2017

KyleAMathews commented Nov 30, 2017

KyleAMathews commented Nov 30, 2017

KyleAMathews commented Nov 30, 2017

PepperTeasdale commented Nov 30, 2017

PepperTeasdale commented Dec 4, 2017 • edited

bjrn left a comment

Choose a reason for hiding this comment

bjrn Dec 7, 2017

Choose a reason for hiding this comment

PepperTeasdale Dec 7, 2017

Choose a reason for hiding this comment

bjrn Dec 7, 2017

Choose a reason for hiding this comment

PepperTeasdale commented Dec 11, 2017

KyleAMathews Dec 12, 2017

Choose a reason for hiding this comment

PepperTeasdale Dec 12, 2017 • edited

Choose a reason for hiding this comment

bjrn Dec 13, 2017

Choose a reason for hiding this comment

PepperTeasdale commented Dec 19, 2017

KyleAMathews commented Dec 19, 2017

KyleAMathews commented Dec 19, 2017

PepperTeasdale commented Dec 20, 2017 • edited

wysher commented Mar 11, 2018

PepperTeasdale commented Mar 11, 2018

camsloanftc commented Mar 20, 2018

PepperTeasdale commented Mar 21, 2018

PepperTeasdale commented Mar 21, 2018

camsloanftc commented Mar 23, 2018

camsloanftc commented Apr 11, 2018

KyleAMathews commented Apr 14, 2018

Khaledgarbaya commented Apr 16, 2018

PepperTeasdale commented Nov 28, 2017 •

edited

gatsbybot commented Nov 28, 2017 •

edited

gatsbybot commented Nov 28, 2017 •

edited

PepperTeasdale commented Dec 4, 2017 •

edited

PepperTeasdale Dec 12, 2017 •

edited

PepperTeasdale commented Dec 20, 2017 •

edited