fix(gatsby-source-contentful): Improve network error handling #30257

axe312ger · 2021-03-15T13:38:57Z

improve sync progress & error logging
reintroduce retry logic from Contentful SDK
allow setting of any value of Contentful SDK again (VERY helpful for tests and debugging aka setting request headers)
add tests

wardpeet · 2021-03-16T11:58:15Z

Can you elaborate on "allow setting of any value of Contentful SDK again (VERY helpful for tests and debugging aka setting request headers)". Why would you want to augment it, instead of adding these values to the plugin itself?

Why don't we support a retry option, ...? You'll also need to update plugin validation in gatsby-node

axe312ger · 2021-03-16T12:08:11Z

@wardpeet: the Contentful SDK allows a lot of options, some might be relevant in rare use cases. Most people won't need this at all. Keeping these in sync with Gatsby configuration is extra work we can avoid. https://github.com/contentful/contentful.js/#configuration

This PR needs it to speed up the tests, otherwise it would retry 30seconds in one test.

Also:

this feature existed before but got broken
I need this to be backported, adding single values is adding new features and we do not add features to the v2 branch
Adding them as options directly to the source plugin will encourage people to play with the values. I don't want them to do that, only people that have experience with the Contentful SDK should set them
the retryLimit ~~is not even documented in the SDK docs :D - the SDK just passes anything down to axios :D~~ is pretty much hidden, the README of the SDK is not up2date. https://contentful.github.io/contentful.js/contentful/8.2.0/contentful.html#.createClient

…sage to the end user

packages/gatsby-source-contentful/src/gatsby-node.js

packages/gatsby-source-contentful/src/fetch.js

Vinz93 · 2021-03-16T14:10:07Z

packages/gatsby-source-contentful/README.md

@@ -181,6 +179,12 @@ Number of entries to retrieve from Contentful at a time. Due to some technical l

 Number of workers to use when downloading Contentful assets. Due to technical limitations, opening too many concurrent requests can cause stalled downloads. If you encounter this issue you can set this param to a lower number than 50, e.g 25.

+**`contentfulClientConfig`** [object][optional] [default: `{}`]
+
+Additional config which will get passed to [Contentfuls JS SDK](https://github.com/contentful/contentful.js#configuration).


Should we add retryLimit in the SDK readme?

Sure, but if you do, pls make sure to add the other potential missing config options there as well :)

Co-authored-by: Ward Peeters <ward@coding-tech.com>

wardpeet

I've added some minor comments, I'm not 100% confident on the pluginOptions part as we don't have tests for it. We can test when we merge this and try out the @next package.

wardpeet · 2021-03-22T09:45:28Z

packages/gatsby-source-contentful/src/fetch.js

+const createContentfulErrorMessage = e => {
+  // If we got a response, it is very likely that it is a Contentful API error.
+  if (e.response) {
+    let parsedContentfulErrorData = {}


we shouldn't set empty object here, null is fine

Suggested change

let parsedContentfulErrorData = {}

let parsedContentfulErrorData = null

wardpeet · 2021-03-22T09:46:42Z

packages/gatsby-source-contentful/src/fetch.js

+        if (typeof parsedContentfulErrorData !== `object`) {
+          throw new Error()
+        }


JSON.parse('string') will throw an error when it's not parsable. Unsure what you're trying to guard against.

wardpeet · 2021-03-22T09:47:24Z

packages/gatsby-source-contentful/src/fetch.js

+
+  let errorMessage = [
+    // Generic error values
+    e.code && `${e.code}`,


Suggested change

e.code && `${e.code}`,

e.code && String(e.code),

packages/gatsby-source-contentful/src/fetch.js

axe312ger · 2021-03-22T15:16:35Z

@wardpeet I (re)implemented pluginOptions that way because I try to avoid adding new features to this PR and just restore old ones.

I need this to be merged into v4 of gatsby-source-contentful and that version has feature freeze due to Gatsby v3 release (has it?)

Alternatively I could directly implement maxRetries and headers and do not allow passing "anything" to Contentful SDK. Would you prefer this?

axe312ger · 2021-03-22T15:18:24Z

Alternative #2:

Remove the pluginOptions completely and let the retry test run for 30+ seconds. The real real reason I re-added it was to make this test take less time: https://github.com/gatsbyjs/gatsby/pull/30257/files#diff-ff025519f6b1f94c06ec41b72834e0426b77069be638c92bbc3eb34a83d1573dR80

wardpeet

LGTM! THank you!

Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4)

#30568) Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4) Co-authored-by: Benedikt Rötsch <axe312ger@users.noreply.github.com>

Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4)

#30617) Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4) Co-authored-by: Benedikt Rötsch <axe312ger@users.noreply.github.com>

axe312ger added 2 commits March 15, 2021 14:36

fix: do not log empty values when logging requests

8b454e9

fix: remove progress total as the real total is not available

3568845

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Mar 15, 2021

fix: allow passing of any option to Contentful SDK again

227754c

axe312ger added status: needs core review Currently awaiting review from Core team member topic: source-contentful Related to Gatsby's integration with Contentful and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Mar 15, 2021

axe312ger added this to In progress in Contentful source plugin improvements via automation Mar 15, 2021

axe312ger marked this pull request as ready for review March 15, 2021 16:33

axe312ger marked this pull request as draft March 15, 2021 18:39

fix: pretty print Contentful errors after all retry attempts failed

8946b74

axe312ger force-pushed the fix/contentful-improve-network-error-handling branch from 495f495 to 8946b74 Compare March 16, 2021 10:22

Merge branch 'master' into fix/contentful-improve-network-error-handling

1226330

docs: add reintroduced contentfulClientConfig to the plugin validation

6da14b6

axe312ger force-pushed the fix/contentful-improve-network-error-handling branch from db291ba to 6da14b6 Compare March 16, 2021 12:20

axe312ger added 3 commits March 16, 2021 13:24

docs: add missing quote

935f1d6

fix: allow error responses with JSON strings

9b03227

fix: do not pass error object to reporter.panic to display custom mes…

f2a6e7b

…sage to the end user

axe312ger marked this pull request as ready for review March 16, 2021 13:41

wardpeet reviewed Mar 16, 2021

View reviewed changes

packages/gatsby-source-contentful/src/gatsby-node.js Outdated Show resolved Hide resolved

wardpeet reviewed Mar 16, 2021

View reviewed changes

packages/gatsby-source-contentful/src/fetch.js Show resolved Hide resolved

packages/gatsby-source-contentful/src/fetch.js Outdated Show resolved Hide resolved

Vinz93 reviewed Mar 16, 2021

View reviewed changes

axe312ger and others added 3 commits March 16, 2021 15:49

style: clean up error message creation function

7f0b22a

Update packages/gatsby-source-contentful/src/gatsby-node.js

2d2dc07

Co-authored-by: Ward Peeters <ward@coding-tech.com>

test: revert change to unrelated snapshot

e013695

wardpeet reviewed Mar 22, 2021

View reviewed changes

revert normalize changes

d4716d5

wardpeet approved these changes Mar 23, 2021

View reviewed changes

axe312ger added bot: merge on green Gatsbot will merge these PRs automatically when all tests passes and removed status: needs core review Currently awaiting review from Core team member labels Mar 23, 2021

wardpeet added this to To cherry-pick in V2 Release hotfixes via automation Mar 29, 2021

wardpeet changed the title ~~Contentful: Improve network error handling~~ fix(gatsby-source-contentful): Improve network error handling Mar 29, 2021

wardpeet merged commit c1ac5e4 into master Mar 29, 2021

Contentful source plugin improvements automation moved this from In progress to Done Mar 29, 2021

wardpeet deleted the fix/contentful-improve-network-error-handling branch March 29, 2021 08:38

axe312ger added this to To cherry-pick in Release candidate via automation Mar 30, 2021

gatsbybot mentioned this pull request Mar 30, 2021

fix(gatsby-source-contentful): Improve network error handling (#30257) #30568

Merged

vladar pushed a commit that referenced this pull request Mar 30, 2021

fix(gatsby-source-contentful): Improve network error handling (#30257)

728adf4

Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4)

vladar moved this from To cherry-pick to Backport PR opened in Release candidate Mar 30, 2021

vladar moved this from Backport PR opened to Backported in Release candidate Mar 30, 2021

gatsbybot mentioned this pull request Apr 1, 2021

fix(gatsby-source-contentful): Improve network error handling (#30257) #30617

Merged

pieh pushed a commit that referenced this pull request Apr 1, 2021

fix(gatsby-source-contentful): Improve network error handling (#30257)

d018bab

Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4)

pieh moved this from To cherry-pick to Backport PR opened in V2 Release hotfixes Apr 1, 2021

pieh pushed a commit that referenced this pull request Apr 1, 2021

fix(gatsby-source-contentful): Improve network error handling (#30257)

1c348da

Co-authored-by: Ward Peeters <ward@coding-tech.com> (cherry picked from commit c1ac5e4)

pieh moved this from Backport PR opened to Backported in V2 Release hotfixes Apr 2, 2021

vladar mentioned this pull request May 18, 2021

fix(gatsby-source-contentful): fix progress bar #31467

Merged

tonyg101 mentioned this pull request Jul 29, 2023

[Snyk] Upgrade gatsby from 2.20.25 to 2.32.13 signal-noise/gatsby#4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(gatsby-source-contentful): Improve network error handling #30257

fix(gatsby-source-contentful): Improve network error handling #30257

axe312ger commented Mar 15, 2021 •

edited

wardpeet commented Mar 16, 2021

axe312ger commented Mar 16, 2021 •

edited

Vinz93 Mar 16, 2021

axe312ger Mar 16, 2021

wardpeet left a comment

wardpeet Mar 22, 2021

wardpeet Mar 22, 2021

wardpeet Mar 22, 2021

axe312ger commented Mar 22, 2021

axe312ger commented Mar 22, 2021

wardpeet left a comment

	let parsedContentfulErrorData = {}
	let parsedContentfulErrorData = null

fix(gatsby-source-contentful): Improve network error handling #30257

fix(gatsby-source-contentful): Improve network error handling #30257

Conversation

axe312ger commented Mar 15, 2021 • edited

wardpeet commented Mar 16, 2021

axe312ger commented Mar 16, 2021 • edited

Vinz93 Mar 16, 2021

Choose a reason for hiding this comment

axe312ger Mar 16, 2021

Choose a reason for hiding this comment

wardpeet left a comment

Choose a reason for hiding this comment

wardpeet Mar 22, 2021

Choose a reason for hiding this comment

wardpeet Mar 22, 2021

Choose a reason for hiding this comment

wardpeet Mar 22, 2021

Choose a reason for hiding this comment

axe312ger commented Mar 22, 2021

axe312ger commented Mar 22, 2021

wardpeet left a comment

Choose a reason for hiding this comment

axe312ger commented Mar 15, 2021 •

edited

axe312ger commented Mar 16, 2021 •

edited