New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setting unlimited maxCapacity, int vs uint #55
Comments
Hey @Kaszanas! Unlimited task queuesThis library doesn't currently have support for setting up unlimited task queues. This was an intentional design decision for the sake of simplicity and efficiency. The task queue is currently implemented as a fixed-size buffered channel, which takes a constant space in terms of memory and minimizes the number of memory allocations when inserting or removing tasks.
That said, I would encourage you to run some stress/load tests to determine a reasonable value for the Int vs Uint?You are right, these options could have been |
Thanks for the swift reply @alitto yeah I only wanted to have an unlimited task queue because it can get filled quicker than processing happens. In that case it could be hard to foresee what should be it's size. |
Background
I have a system where I will be using pond for network related download tasks.
To avoid hitting rate limiters when issuing multiple network requests it would be great to be able to set unlimited task capacity.
Imagine a scenario where there are some downloads that are already finished and the data is easily accessible so there is no need to issue another download.
In such case a broader worker pool can have as much goroutines as number of CPU threads. But then you wouldn't like to have that for a worker pool downloading because you'd have potentially 24 - 56 threads attempting that at once.
Question
Is there a way to have unlimited tasks queue and just 3-5 workers downloading at the same time? I was not able to find such example in the documentation and the code.
Would this mean that it should be set to the maximum integer capacity?
Int vs Uint?
Additionally given that these values cannot be lower than zero:
Wouldn't it be better to have these types be
uint
?The text was updated successfully, but these errors were encountered: