[proposal] http.requests

[mrmlnc]

Description

Library for working with network requests.

Motivation

Debug network requests without logs is an another art form. Most likely you won't be able to say what happened here, because you or your colleague forgot to add enough debug information. Also network requests can have retries due to network or service problems that will not be displayed in the logs. This is critical problem in teams that work with automated app releases, money, or large and complex processes.

Goals

• Logging (injectable logger) of each stage of the network request lifecycle.
  • request
  • response
  • redirect
  • retry-planned
  • retry-skipped
• Ability to log parts of request
  • showQueryFields (boolean | string[]) — to control the display of Query parameters
  • hideQueryFields (string[]) — to hide some specific Query parameters like token or something else
  • showPayloadFields (boolean | string[]) — to control the display of Payload fields only for JSON payload type
  • hidePayloadFields (string[]) — to hide some specific Payload fields
  • truncateResponseBodyAfter (number) — to limit the message length for failed requests in log
  • requestId (string) — ability to set custom request id (used for x-request-id)
  • ...
• Mark each request by x-request-id header. The request id does not change during a single request lifecycle (request → response → retry-planned → …).
• Provide an ability to control retires by function.

Details

Based on got. Fully compatible API without additional hacks. At least for now.

Log

GET request

{
	"type": "request",
	"id": "c781e1c2a07805a3f7f359473a85adaf",
	"method": "GET",
	"url": "http://canonium.com/pathname",
	"query": {
		"author": "robot",
		"status": "active",
		"page": "1",
		"page_size": "100"
	}
}

GET redirect

{
	"type": "redirect",
	"id": "c781e1c2a07805a3f7f359473a85adaf",
	"method": "GET",
	"url": "http://canonium.com/pathname",
	"to": "https://canonium.com/pathname"
}

POST request

{
	"type": "request",
	"id": "c781e1c2a07805a3f7f359473a85adaf",
	"method": "POST",
	"url": "https://canonium.com/pathname",
	"payload": {
		"title": "<value>",
		"body": {
			"<key>": "<value>"
		}
	}
}

GET response

{
	"type": "response",
	"id": "c781e1c2a07805a3f7f359473a85adaf",
	"method": "GET",
	"url": "https://canonium.com/pathname",
	"status": 404,
	"message": "Not Found",
	"code": "ENOTFOUND", // For network problems
	"body": "Some text of the response that can be truncated by limit<truncated>"
}

GET retry-planned

{
	"type": "retry-planned",
	"id": "c781e1c2a07805a3f7f359473a85adaf",
	"method": "GET",
	"url": "https://canonium.com/pathname",
	"retryNumber": 1,
	"maxRetryNumber": 4,
	"nextRetryDelay": 2059,
	"maxRetryDelay": 60000,
	"nextRetryDate": "2020-10-05T06:40:51.454Z",
	// Something else?
}

GET retry-skipped

{
	"type": "retry-skipped",
	"id": "c781e1c2a07805a3f7f359473a85adaf",
	"method": "GET",
	"url": "https://canonium.com/pathname",
	"reason": "Skipped due to non-retriable status code: 404",
	// Something else?
}

[mrmlnc]

Some background work around this issue.

• Set default value for an options object sindresorhus/got#1495

Blocked by sindresorhus/got#1443, because we want to use the context field for our own properties like logger, request.id and options.*.

[mrmlnc]

https://nodejs.org/en/blog/announcements/v18-release-announce/#fetch-experimental