HTTPError: Response code 422 (Unprocessable Entity) upon second execution of pagination.paginate #1221
Comments
|
The HTTP status hint at a malformed request, despite the |
|
I was able to validate that the 422 is indeed the response from the remote rest api server. Now given that all is fine when using Postman and that the |
|
What's your |
|
I may have something to start looking into. First let's consider this "I will cause a 422 on second page" snippet: const iterator = client.paginate<Account>('Accounts', {
json: {
fields: 'name,siret_c',
filter: [{ $and: [{ siret_c: { $not_null: null } }, { siret_c: { $not_equals: '' } }] }],
},
});
for await (const account of iterator) {
// Code in here is not relevant to the issue
}Now let's change the const iterator = client.paginate<Account>('Accounts', {
body: JSON.stringify({
fields: 'name,siret_c',
filter: [{ $and: [{ siret_c: { $not_null: null } }, { siret_c: { $not_equals: '' } }] }],
}),
});
for await (const account of iterator) {
// Code in here is not relevant to the issue
}And I now get a different error (after correctly processing the first page) Or is it me missing something obvious? :/ |
|
@szmarczak Here is the {
method: 'GET',
retry: {
calculateDelay: [Function: calculateDelay],
limit: 2,
methods: [ 'GET', 'PUT', 'HEAD', 'DELETE', 'OPTIONS', 'TRACE' ],
statusCodes: [
408, 413, 429, 500,
502, 503, 504, 521,
522, 524
],
errorCodes: [
'ETIMEDOUT',
'ECONNRESET',
'EADDRINUSE',
'ECONNREFUSED',
'EPIPE',
'ENOTFOUND',
'ENETUNREACH',
'EAI_AGAIN'
],
maxRetryAfter: Infinity
},
headers: {
'user-agent': 'got (https://github.com/sindresorhus/got)',
'oauth-token': 'd5179e7e-b9f5-49ed-9330-ab4b9e0d1f27',
'content-type': 'application/json',
'content-length': '109',
'accept-encoding': 'gzip, deflate, br'
},
cache: undefined,
dnsCache: CacheableLookup {
maxTtl: Infinity,
fallbackTtl: 1,
errorTtl: 0.15,
_lockTime: 150,
_cache: Map { 'myHostedInstance.myHostingProvider.com' => [Array] },
_resolver: Resolver { _handle: ChannelWrap {} },
_lookup: [Function: lookup],
_resolve4: [Function: bound queryA],
_resolve6: [Function: bound queryAaaa],
_iface: { has4: true, has6: true },
_hostsResolver: HostsResolver {
_hostsPath: '/etc/hosts',
_error: null,
_hosts: [Object],
_promise: null
},
_tickLocked: false,
_pending: {},
lookup: [Function: bound lookup],
lookupAsync: [Function: bound lookupAsync] AsyncFunction
},
decompress: true,
throwHttpErrors: true,
followRedirect: true,
isStream: false,
responseType: 'text',
resolveBodyOnly: false,
maxRedirects: 10,
prefixUrl: [Getter/Setter],
methodRewriting: true,
ignoreInvalidCookies: false,
http2: false,
allowGetBody: true,
rejectUnauthorized: true,
pagination: {
transform: [Function: transform],
paginate: [Function: paginate],
filter: [Function: filter],
shouldContinue: [Function: shouldContinue],
countLimit: Infinity,
requestLimit: 10000,
stackAllItems: true
},
username: '',
password: '',
agent: {
http: Agent {
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 80,
protocol: 'http:',
options: [Object],
requests: {},
sockets: {},
freeSockets: {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
[Symbol(kCapture)]: false
},
https: Agent {
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 443,
protocol: 'https:',
options: [Object],
requests: {},
sockets: {},
freeSockets: {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
maxCachedSessions: 100,
_sessionCache: [Object],
[Symbol(kCapture)]: false
}
},
url: URL {
href: 'https://myHostedInstance..com/rest/v11_5/Accounts',
origin: 'https://myHostedInstance.myHostingProvider.com',
protocol: 'https:',
username: '',
password: '',
host: 'myHostedInstance.myHostingProvider.com',
hostname: 'myHostedInstance.myHostingProvider.com',
port: '',
pathname: '/rest/v11_5/Accounts',
search: '',
searchParams: URLSearchParams {},
hash: ''
},
mutableDefaults: true,
hooks: {
init: [],
beforeRequest: [],
beforeRedirect: [],
beforeError: [],
beforeRetry: [],
afterResponse: [ [AsyncFunction], [AsyncFunction] ]
},
lookup: [Function: bound lookup],
searchParams: undefined,
request: undefined,
timeout: {},
[Symbol(isNormalizedAlready)]: true,
[Symbol(request)]: [Function: request]
} |
|
Can you try the following:
If any of these are don't fix the issue, please post your pagination function and after response hooks (there are two, right?)
That definitely seems to be a bug. It shouldn't throw the |
I tried each and both without any improveemnt.
Those might seem familiar for I made you sweat on it last week ;) pagination: {
// eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
paginate: (response: Response) => {
const body: List = JSON.parse((response as Response<string>).body);
if (is.plainObject(body) && is.number(body.next_offset) && body.next_offset >= 0) {
const { options } = response.request;
const { json, ...otherOptions }: any = options;
const result: Options = {
json: { ...json, offset: body.next_offset },
...otherOptions,
};
return result;
}
return false;
},
// eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
transform: (response: Response<any>) => {
const { body } = response;
const parsed = is.string(body) ? JSON.parse(body) : body;
if (is.plainObject(parsed)) {
const { records } = parsed;
if (is.array(records)) {
return records as unknown[];
}
throw new Error(typeof records);
}
throw new Error(typeof parsed);
},
}, afterResponse: [
// eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
async (response, retryWithMergedOptions) => {
// Unauthorized
if (response.statusCode === 401) {
const updatedOptions: ExtendOptions = { headers: { 'oauth-token': await getNewAuthToken() } };
// Save for further requests
authenticatedInstance.defaults.options = got.mergeOptions(
authenticatedInstance.defaults.options,
updatedOptions
);
return retryWithMergedOptions(updatedOptions);
}
// No changes otherwise
return response;
},
],(On my TODO list is a cleaner, more elegant and efficient management of OAuth2 token, including effective renewal)
I feel like I am being a pain here... ;) |
I haven't been able to work on Got that much today. But will do ASAP when I have the time. I see that you're passing a Also try logging
No, you just find very interesting bugs, which is awesome. By contributing to Got you make it even better. Here you can see two after response hooks. But in the code you provided there's only one. Another bug here? |
Yeah... I forgot... Hit me :/
The value of {
"fields": "name,siret_c",
"filter": [
{
"$and": [
{
"siret_c": {
"$not_null": null
}
},
{
"siret_c": {
"$not_equals": ""
}
}
]
}
],
"offset": 20
} |
|
@szmarczak I have an idea of what could be causing the issue. Between first and second requests, the size of Anyhow, in Postman, truncating the json payload (hence sending invalid json) is one sure way to get a 422 response. |
|
@szmarczak My idea seems to be correct. I switched to using Thus I deduce that somehow the payload does indeed ends up being truncated upon requesting the second page. I hope this will help. |
I need your input on this too :) |
Guess what? I have one single Now if I inspect Funny for I was about to add my share to #1220 . |
szmarczak
added
bug
|
@szmarczak Here is a test case for this bug // Nicer test case in PR #1231 The relevant part is |
|
Thanks! I will definitely check this. |
|
Don't do |
szmarczak
added
unconfirmed bug
|
I fixed your test 47c1afe and it's not failing... |
|
Can you try this: paginate: (response: Response) => {
const body: List = JSON.parse((response as Response<string>).body);
if (is.plainObject(body) && is.number(body.next_offset) && body.next_offset >= 0) {
const { options } = response.request;
const { json } = options;
const result: Options = {
json: { ...json, offset: body.next_offset }
};
return result;
}
return false;
}, |
I think that's it. Let me check. |
|
I'm unable to reproduce even with 5131dc2 I'll make a release now. |
|
Using Got@11.1.3 without the 'fixed sized |
|
#1221 (comment) Did you try it? |
|
So... this is in fact another symptom of #1223 root cause. As soon as having Thanks for your patience and time @szmarczak :) |
Describe the bug
Iteration records from SugarCRM rest api with Got.paginate()
Actual behavior
After processing the first 20 records (default page size in the api), an HTTP 422 error is thrown:
I have inspected the
normalizedOptionsatnode_modules/got/dist/source/create.jsline 148 and saw nothing wrong. I also validated that the headers & json payload were fine by shooting Postman. Last but not least, agent's keepAlive being true or false has no impact.Expected behavior
Got.paginate() should process more than one page.
Code to reproduce
Checklist
The text was updated successfully, but these errors were encountered: