Optimizing batch requests

[martinAlt335]

Hi, just had a few questions about Undici and optimizing for running a tonne of requests.

I have a batch of 10k HTTP requests to make and want to squeeze-out every ounce of juice I can get. In the test below I purposefully set connections to 10,000 to see what would happen.

const isRejectedGuard = <T>(
  input: PromiseSettledResult<T>,
): input is PromiseRejectedResult => input.status === 'rejected';

const connections = 10000;
const dispatcher = new Pool(new URL({URL_HERE}), {
  connections
});
const arr = Array.from(Array(connections).keys());
let startAt: bigint;
let endAt: bigint;
let counter = 0;

async function testClient() {
  startAt = process.hrtime.bigint();
  return Promise.allSettled(
    arr.map(() => {
      counter++;
      if (counter === arr.length) {
        console.log(
          "Connected",
          (dispatcher as any).stats.connected,
          "free",
          (dispatcher as any).stats.free,
          "queued",
          (dispatcher as any).stats.queued,
          "pending",
          (dispatcher as any).stats.pending,
          "size",
          (dispatcher as any).stats.size,
          "running",
          (dispatcher as any).stats.running
        );
      }
      return dispatcher.request({
        path: "{{PATH_HERE}}",
        method: "POST" as HttpMethod,
        body: JSON.stringify({
          jsonrpc: "2.0",
          id: 1,
          {{PARAMS_HERE}}
        }),
        headers: ["content-type", "application/json"],
      });
    })
  ).then((r) => {
    endAt = process.hrtime.bigint();
    return r;
  });
}

(async function () {
  await testClient().then((r) => {
    console.log(
      `Benchmark stats:`,
      "\n",
      "Took",
      +((endAt - startAt) / BigInt(1000000)).toString(),
      "ms",
      "\n",
      arr.length,
      "requests total.",
      "\n",
      r.filter(isFulfilled).length,
      "completed",
      "\n",
      arr.length - r.filter(isFulfilled).length,
      "failed",
      "\n",
      connections,
      "connections"
    );
    const data = JSON.stringify(r.filter(isRejectedGuard).map((q) => q.reason));
    fs.writeFile("errors.json", data, () => {});
  });
})();

Running the above took a total of 38960 ms, of which 4,683 requests completed and 5,317 failed. Here's a full picture of the run stats;

The errors.json file has the same failing error for each failed request (also like to note this error is not documented in the errors types file);

{"name":"ConnectTimeoutError","code":"UND_ERR_CONNECT_TIMEOUT","message":"Connect Timeout Error"}

I am aware about file descriptor limitations on Linux and Windows handler limitations (this test was ran on a Windows machine), and port exhaustion though I checked my machines event logs for port exhaustion and none were reported, also don't believe I ran into a handler limitation.

Here's a recording of the sockets lifecycle captured for this run (partially for the hell of it, and if it helps at all with seeing the socket lifecycle states / relating it to the failed request error. Some states may possibly be missed given TaskExplorer itself is set to refresh every 300ms) (sockets action starts at 0:09);

sockets_lifecycle.mp4

I'd just like to ask

• a) how I can improve this process, the goal is to make these 10k requests as fast as possible, granted the majority should be succeeding ;) I'm not entirely sure as to what's causing this timeout? Could it be the endpoint I'm testing against? I saw the mock classes with Unidici, would it be worth attempting to return a mocked response for this request and observe?
• b) about any 'gotchas' / limitations to be aware-of when attempting to do such a task.
• c) just a general question; if forking to a child process to say invoke this function; meaning there is a possibility of 20k reqs being executed at a given point in time, would there be any more things to watch out for apart from literal hardware limitations?

[mcollina]

The most usual case is the lack of Operating System resources to perform those requests in parallel. You are asking your OS to open 10k sockets, each one consuming a file descriptor and a few KB of memory. If each one of them needs 64KB (very common), you'll need a bit more than 600MB of RAM in kernel-space to be able to allocate them. Essentially such high number of concurrent connections are possible only with sophisticated network tuning.
Note that you definitely need a lot of memory in Node.js too because you are creating 10000 TLS contexts.

Given how TCP windows work, I recommend to create a fixed number of connections, e.g. 128, and then set up pipelining to maximize their throughput, e.g. 8 or 10 pipelined requests.

[martinAlt335]

Right on, makes sense. I wonder who exactly was the culprit causing said requests in the test bench I ran above to fail?

Would greatly appreciate any pointers to read up on in regards to increasing available throughout. Do you have any specific points / areas of concern (say on a Node, OS, etc, level) I can focus on to push to higher levels of concurrency?

These requests are POST requests given they are JSON-RPC 2 spec'd. Will pipelining GET requests with an application/json body attached still work? I recall attempting to encode the method as outlined by the spec when using GET requests but the endpoint refused to accept an empty request body. I was able to make the GET request with the request body attached.

Cheers!

[mcollina]

This is a good guide: https://cromwell-intl.com/open-source/performance-tuning/tcp.html.