Dask in place of celery for performing background tasks in flask

[billlyzhaoyh]

I have a web app developed in flask that currently uses dask distributed to execute tasks that are essentially solving differential equation with scipy ode

My question is about this portion of documentation on dask distributed describing as_completed call on a group of futures:

"For greater efficiency, you can also ask as_completed to gather the results in the background:
for future, result in as_completed(futures, with_results=True): # y = future.result() # don't need this ...
"

Does this mean when I try to collect the futures back from the distributed client it will be non-blocking? As I am running flask server locally without multi threading and my application does not appear to hand while the local cluster is computing the tasks.

[mivade]

Does this mean when I try to collect the futures back from the distributed client it will be non-blocking?

No. Flask (and WSGI in general) is still synchronous so if you wait until the tasks are complete, the request will be pending until then. In order to implement non-blocking tasks with Flask you would need to mimic what is usually done with task queues like Celery: return some ID to the client as soon as a future is created and include an additional endpoint for polling.

[billlyzhaoyh]

Thanks for the comment so far Mike and that is really helpful.

I have never used Dask in a web app context before so I am struggling to wrap my head around it when it comes to choosing a framework for task execution (celery vs dask). The parallel computing features of Dask is pretty powerful and in the first stage, the web app will only serve a handful of users.

For context, we are doing a volunteering project where we will receive some input from the users regarding some refugee camp parameters and we will process those for epidemiology modelling and the running of models is processes as tasks (differential equations, matrix manipulations, network type of calculations etc). There is plenty of material out there for flask+celery but not much for flask+dask so that is why I am reviewing whether we should press ahead with Dask (we are getting roughly 10x time saved by parallel execution in Dask) In celery it becomes a matter of spinning up more workers in a machine.

What do you think I should do in this situation?

[mrocklin]

Briefly, this example using Dask with Tornado in an async context may or may not be useful to you (probably not, given the focus on flask)

https://examples.dask.org/applications/async-web-server.html

[mivade]

Whether you're using Celery or Dask the problem of blocking is the same when using a WSGI framework like Flask. The actual computation can happen asynchronously in either case but at the end of the day you want to provide results. What do your outputs look like and how big are they? For the sake of argument, say that when all your tasks are complete you end up with some JSONizable data that is on the order of 100 kB. That's a pretty manageable size and I wouldn't think twice about fetching it from the cluster and returning as a response. If instead you have some binary data on the order of MB or more then you'd probably want to come up with a different solution (using an asynchronous framework and chunking the response, providing a link to an S3 or similar URL, etc.).

[billlyzhaoyh]

right gotcha - the data are more than 100 Mb so that might pose a problem. My current thinking is to write the results to a DB after the run (but parallel writing to DB is blocked...). I started looking at celery because this post https://stackabuse.com/asynchronous-tasks-using-flask-redis-and-celery/ it seems like once the tasks are pushed into the queue and since they are run in a different worker that is kind of Async?

Fastapi like frameworks are great but current we use Dash which uses a WSGI framework and we have gone quite far down the flask train :)

[mivade]

For large datasets I think probably the best option is to just transfer the results to some location that can be served by nginx or some other server that's optimized for static file serving. Your Dash app can provide a link to the user when the results are ready. I would do this even if I were using Tornado, FastAPI, or whatever other async framework because I don't want to be the one responsible for tuning the chunking parameters correctly :)

it seems like once the tasks are pushed into the queue and since they are run in a different worker that is kind of Async?

Yeah, the term "async" is a little overloaded. The asynchronous bit when using Celery or Dask is that you're requesting a task to launch but aren't getting the results back until later. That's distinct from "async" as in async def coroutines.