Replies: 1 comment 2 replies
-
When I need to do this I tend to pass along some token between the different tasks @dask.delayed
def task1():
pass
@dask.delayed
def task2(extra=None):
pass
@dask.delayed
def task3(extra=None):
pass
@dask.delayed
def token(x):
return None
t1 = task1()
t2 = task2(extra=token(t1))
t3 = task3(t2, extra=token(t1)) We don't pass any real data (just None) and it's easy enough to reuse the existing system. It's not exactly what you want, but it might do |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
As far as I'm aware you can only apply dependencies that are inherently implied through a functions arguments. It is potentially contrary to the computational nature of the dask library, but there could be (and are depending on how you use it) cases where you would want to depend on a task for its side-effects, rather than return value.
You can see this suggestion in action in the
aiodag
library. This library provides an interface heavily influenced by dask.delayed to implicitly build dags powered by asyncio.https://github.com/aa1371/aiodag#endogenous-vs-exogenous-depedencies
An example of the interface extrapolated to dask.delayed is as follows:
In this case task1 is applied as a dependency to task2 and task3, even though neither expects or uses its result.
I briefly tried to search for any related issues open or closed, but was quite a bit to digest and couldn't find anything. Would there be any appetite for this kind of (or some similar) interface?
Beta Was this translation helpful? Give feedback.
All reactions