[QUESTION] How to make parallel / multiple requests at once?

[higoka]

I have an array of files (10'000+ files) which i need to download from the server. Im not sure how i can implement so that the requests so they don't block each other like make one request -> wait until downloaded -> start next request.

Is it possible to somehow download all the files in parallel / multiple requests to increase the performance?

also is it possible to keep the connection alive since all the requests are made to the same server?

[sindresorhus]

Your question is not directly related to Got, but it's more of a general JavaScript question. Got returns a promise. There are many ways to run promises concurrently. I would recommend using something like p-map as it lets you limit the concurrency. What the concurrency limit should be really depends on the server. You have to make sure you won't get banned by the server owner.

[higoka]

Is it possible to use Promise.allSettled with a custom http agent which has the maxSockets set to something like 100. Is this equivalent of using p-map ?

[xairoo]

@higoka any updates you can share?

p-queue, p-map works, but comparing the results with autocannon isn't great. I am looking for a solution to crawl many sites periodically. I talk about 10.000 request per second / thread.

[higoka]

Im using Promise.allSettled() with maxSockets but im not sure if this is the right way of doing it. I also dont know a way to test it?

[xairoo]

fastq is amazing. Try this:

import { promise as fastq } from 'fastq';
import got from 'got';

async function stall(stallTime = 1000) {
  await new Promise((resolve) => setTimeout(resolve, stallTime));
}

const queue = fastq(worker, 5); // 5 parallel requests

let i = 1;
while (i <= 10) {
  queue.push({ i, url: 'http://google.com' });
  i++;
}

async function worker(obj) {
  //   await stall(); // Make it more "visible" for the demo
  const { statusCode } = await got(obj.url);
  console.log({ i: obj.i, date: new Date(), statusCode });
}