Best way to implement "retry on 404"

[PopGoesTheWza]

What would you like to discuss?

For internal business use, I interact with a closed source, third party Restful API. Performances are good but stability was very fine until recently.

In a nutshell, this Rest API server sometimes get into a confused state and returns an http 404 response even though the request is valid and result exists.

We currently identified 2 workarounds:

1. do nothing! the server will get its mind together in a few minutes and it'll all be fine later.
2. issue a 'reset connection pool' http query and resubmit the request. From our preliminary tests, that option always fix the issue (until it occurs again in a few hours)

To implement the second workaround, an afterResponse hook seems to be the way to go. Something roughly like the following

{
    afterResponse: [
      async (response: Response, retryWithMergedOptions) => {
        if (response.statusCode === 404) {
          const updatedOptions: ExtendOptions = {};

          await resetConnectionPool(); // restore API server

          return retryWithMergedOptions(updatedOptions);
        }

        // No changes otherwise
        return response;
      },
    ],
  }

Yet I am unsure how to best implement a "retry on 404 only once" rule, to prevent the unlikely/rare occasions where 404 is the correct response.

Thus my questions are:

• is an afterResponse hook indeed the best way to implement this workaround?
• any advise on how to manage a state or "retry" counter so that the workaround can only be triigered once per request?

TIA

Checklist

• I have read the documentation.

[szmarczak]

To manage the state you can use context

[PopGoesTheWza]

@szmarczak is it me or the afterResponse hook is not triggered by the retryWithMergedOptions function? That aspect is not clearly stated within the documentation. This seems to be the case from the tests I did (using got 11.x)

If this is WAD, then there is not need to try to maintain and check the number of 'resetConnectionPool' calls, since the afterResponse hook is invoked only once per request.

[szmarczak]

All hooks that are executed prior to retryWithMergedOptions will be skipped in the next retries. This it to prevent infinite loops. The same happens with Got v12.

A PR is welcome :)

[PopGoesTheWza]

Brilliant! Hence I will re-read the hooks documentation and submit a PR if it needs clarification.

[PopGoesTheWza]

Given that afterResponse is not triggered by retryWithMergedOptions, the solution is in fact straightforward and dead simple:

  const defaultOptions: ExtendOptions = {
    hooks: {
      afterResponse: [
        async (response, retryWithMergedOptions) => {
          const {statusCode} = response;

          if (statusCode === 404) {
            const reset = await resetConnectionPool();
            console.error('!!!', 'resetConnectionPool', reset.statusCode);
            return retryWithMergedOptions({});
          }

          return response;
        },
      ],
    },
    pagination: {paginate, transform, stackAllItems: false},
    responseType: 'json',
    ...authOptions,
  };

  return got.extend(defaultOptions);

[ruslanguns]

Given that afterResponse is not triggered by retryWithMergedOptions, the solution is in fact straightforward and dead simple:

  const defaultOptions: ExtendOptions = {
    hooks: {
      afterResponse: [
        async (response, retryWithMergedOptions) => {
          const {statusCode} = response;

          if (statusCode === 404) {
            const reset = await resetConnectionPool();
            console.error('!!!', 'resetConnectionPool', reset.statusCode);
            return retryWithMergedOptions({});
          }

          return response;
        },
      ],
    },
    pagination: {paginate, transform, stackAllItems: false},
    responseType: 'json',
    ...authOptions,
  };

  return got.extend(defaultOptions);

Amazing, but I am wondering how would this approach be in a different scenario:

There is an external API which has a access_token, and after some time the
AC expires, so we need to retry after requesting a new AC in parallel.

The idea is that, when external API throws a 401 error, instead of throwing an error without the data output, we could get the new AC by performing a login via a HTTP request and then with the new AC (refreshed token) retry the request and send the data output.

What would you recommend? Is it possible?

[szmarczak]

after some time the AC expires, so we need to retry after requesting a new AC in parallel.

It doesn't solve the case when the refresh request times out and you'll get 401 unauthorized again.

[PopGoesTheWza]

What would you recommend? Is it possible?

Isn't the example in hooks.afterresponse precisely about that?

@ruslanguns if you start a discussion for your use case, I'll gladly post the code I use to handle expired token.

[ruslan-ncg]

@PopGoesTheWza I just didn't wanted to duplicate discussion, Will create that now! thanks !

[ruslan-ncg]

I have created #1840 to continue with this subject, thanks in advance!