You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, currently i have two cheap servers with 8 cores each and 64GB of RAM. I thought this would be enough to process at least 100 images a minute but this is not the case.
So I am asking the community to help me out with finding out good hardware/server setup so I can achieve at least 100 processed images per minute.
I am building the image processing layer on top of IPFS and pre-processing each image in 5 different versions, all images are in avif format, and widths are 80, 500,1024, 2056, and 4000 without any added sharpness or blur, and metadata included.
Any help is appreciated.
the output of lscpu:
The text was updated successfully, but these errors were encountered:
pre-processing each image in 5 different versions, all images are in avif
The bottleneck will almost certainly be AVIF encoding (via libaom).
At a guess the problem may be too many (hyper-)threads. You'll need to provide a complete, standalone code sample and images that allows someone else to both reproduce and profile if you'd like help optimising this further.
Unfortunately, I cannot extract the example from the source code, it is not public. But the fact that the avif might be the slow part is already a start.
I assume jpg is faster?
At a guess the problem may be too many (hyper-)threads
Yes, usually significantly, the degree to which can depend on the type of images you're processing.
At a guess the problem may be too many (hyper-)threads
What do you mean by this?
Without sample code and images to reproduce it's hard to say, but it's possible the overhead of thread management and/or CPU cache thrashing might be slowing things. Sometimes reducing concurrency and/or parallelism can help improve throughput.
Hi, currently i have two cheap servers with 8 cores each and 64GB of RAM. I thought this would be enough to process at least 100 images a minute but this is not the case.
So I am asking the community to help me out with finding out good hardware/server setup so I can achieve at least 100 processed images per minute.
I am building the image processing layer on top of IPFS and pre-processing each image in 5 different versions, all images are in
avif
format, and widths are80, 500,1024, 2056, and 4000
without any added sharpness or blur, and metadata included.Any help is appreciated.
the output of
lscpu
:The text was updated successfully, but these errors were encountered: