dagster-mlflow tracking config not used from run_config when already configured in definitions

[ion-elgreco]

Dagster version

1.7.5

What's the issue?

When mlflow_tracking is already configured in the definitions, then it's not possible to override the config through the "run_config". This is a quite unfortunate bug since if you don't provide some dummy config at definitions level, auto-materialize and all assets/jobs won't run due to this missing config.

In my case, I built an asset that allows you to do grid search on hyperparam search space that will trigger multiple runs, and these runs will use a new mlflow config, that are provided in the config of the grid_search_asset, see here: https://github.com/ion-elgreco/dagster-ml/blob/main/src/grid_search_asset_generator.py

What did you expect to happen?

Run_config should be used when passed. As it does on other resources that are inheriting ConfigurableResource

How to reproduce?

You can clone this repo: https://github.com/ion-elgreco/dagster-ml/tree/main

Configure the mlflow_tracking in the definitions, and try to override the config now in the launchpad, you will see it's not picking up the new config

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[ion-elgreco]

@smackesey tagging you :), since your name is popping up for this integration

[ion-elgreco]

Resolved it by creating a custom config Class, https://github.com/ion-elgreco/dagster-ml/blob/7f8dc988f04ec7dd843b69386abde5f2b33e8b20/src/resources.py#L49

Thanks for the tip about ConfigurableResource @danielgafni :)