Parallel jobs running concurrently

[CedricHadjian]

I came across this package today in the morning and I loved its features. I'm working on a software that should run async tasks in parallel using proxies, i.e each proxy would be a job and these proxies would be working on bots and making separate async calls. What do you recommend I look into? We are also planning on using taskforce.js, what are the main differences between taskforce.js and bull-board?

[manast]

Maybe you can get some inspiration and examples on real use cases here: https://blog.taskforce.sh/

[CedricHadjian]

@manast I would really appreciate it if I could get some help with what I should be using to achieve what I want, also, if there's a way to receive premium support, I would also appreciate it. Thanks.

[CedricHadjian]

@manast This is a good link that demonstrates my request: #1752, where can I find a good guide for it?

[manast]

Ok, so one proxy can only be used to make calls to the "bots" one at a time, not in parallel right? And how many proxies would you have in total? a small static number as 2 or a large dynamic number?

[manast]

Anyway, one possible solution:
Just add one job to the queue for every proxy, in the job data store the "used" number and the proxy ID, would be 0 when you add the job initially.
Then in the processor use the proxy ID to perform the call with the proper proxy. If the call succeeds increase the used number and add a new job to the queue with the same proxy ID and the new used number. If the used value gets to 720 you just do not add the job to the queue.
With the above scheme you can have as many proxies as you want and as many processors you want running in parallel, still a given proxy ID will only process one job at a given time.

[CedricHadjian]

Ok, so one proxy can only be used to make calls to the "bots" one at a time, not in parallel right?

Right

And how many proxies would you have in total? a small static number as 2 or a large dynamic number?

A large dynamic number