How to make asset function when its upstream asset is not fixed

[Riku-Sasaki]

Hi, I'm new to Dagster and have a question about asset functions.

In my program, a part of asset flow is not determined in advance and is defined dynamically at runtime.

Following scripts shows an example of my program.

1. split_actions randomly materialize either split_actions_1 or split_actions_2.
2. The materialized asset (split_actions_1 or split_actions_2) is transformed into another asset, downstream_1 or downstream_2. downstream_1 and downstream_2 never exist at the same time.
3. common_actions get the result of step2, downstream_1 or downstream_2, as argument and transform into the final asset.

@asset
def upstream():
    return 0

@multi_asset(outs={"split_actions_1": AssetOut(is_required=False), "split_actions_2": AssetOut(is_required=False)})
def split_actions(upstream):
    if random.randint(1, 10) < 5:
        yield Output(value=upstream + 1, output_name="split_actions_1")  # 1
    else:
        yield Output(value=upstream + 10, output_name="split_actions_2")  # 10

@asset
def downstream_1(split_actions_1):
    return split_actions_1 + 1  # 2

@asset
def downstream_2(split_actions_2):
    return split_actions_2 + 10  # 20

@asset
def common_actions(???):
    # ??? is either downstream_1 or downstream_2
    return ??? + 1  # 3 or 21

How should I make common_actions function to realize this flow?
If you have any good ideas, please let me know.

[garethbrickman]

Edit: removed my erroneous example to prevent future confusion/false positives

[Riku-Sasaki]

Hi @garethbrickman, thank you for your response.

Though I tried making common_actions as you said, it still doesn’t work well.
common_actions seems to be materialized only if both downstream_1 and downstream_2 are materialized successfully.

[garethbrickman]

Ah my apologies, this is the expected behavior that common_actions won't be executed without all its upstream dependencies having been materialized. There are two existing issues about this #19403 and #18667 that duplicate this issue so I'll close it.

Some workaround ideas are to use the materialize method to bypass the skipped state of the un-materialized upstream dependency, or yield a result from @multi_asset even if the asset doesn't return any value.