unroll an array of values for dvc repro

[Rusteam]

When we run foreach .. do ... for items in an array inside dvc.yaml , we might want to join results of each do into one.
It would be nice to be able to unroll the array instead of adding each item separately.

Now I do this:

vars:
  - datasets:
     - one
     - two
     - three

stages:
  prep:
    foreach: ${datasets}
    do:
      cmd: python main.py prep ${item}
      outs:
       - data/processed/{item}
  join:
    cmd: python main.py merge ${datasets[0]} ${datasets[1]} ${datasets[2]}
    deps:
      - data/processed/${datasets[0]}
      - data/processed/${datasets[1]}
      - data/processed/${datasets[2]}

Obviously this leads to errors when we forget to update join stage after updating datasets variable.

We can have an option to unroll an array like this:

join:
    cmd: python main.py merge ${...datasets}
    deps:
      -  data/processed/

[skshetry]

We don't want to complicate parametrization. It looks like what you are looking for is
Incremental processing or streaming in micro-batches #5917.

[dberenbaum]

Hi @Rusteam! This issue seems closely related to #6107. It also shows how this is a difficult problem since you expect a different output from "unrolling the array" than in that issue. If that's the case, would you mind commenting on that issue and we will mark this one as a duplicate?

[Rusteam]

Hi @Rusteam! This issue seems closely related to #6107. It also shows how this is a difficult problem since you expect a different output from "unrolling the array" than in that issue. If that's the case, would you mind commenting on that issue and we will mark this one as a duplicate?

sure let me do that, thanks