Pools for Dags #38920
Unanswered
gianfrancodemarco
asked this question in
Q&A
Pools for Dags
#38920
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
I have a set of related DAGs that are all scheduled to start at the same time (they manage ETL processes). Each of these can also take some time to complete (30 minutes to 1 hour). This also causes them to be delayed, because they have to wait for the workers to free up (I'm okay with that).
Based on my current configuration, they max out my concurrency capacity and block other DAGs from running.
I think the ideal solution for this would be to have something similar to pools but for DAGs.
I'd like to be able to assign them to a group/pool (e.g. "ETL" pool), and set that at most 2 "ETL" DAGs can run in parallel, without changing the parallelism settings for the overall system.
So, I can have e.g. 8 DAGs running in parallel, but at most 2 from the "ETL" pool.
Is there a way to do this already? Or would this be something worth implementing?
Thanks
Beta Was this translation helpful? Give feedback.
All reactions