How to set a parameter within RunConfig to required

[eliu-fracta]

for the code below, if I set CleanedDataConfig's file_path to ..., it will error out. How can I properly configure it so that the job will require file_path as an input?

class CleanedDataConfig(Config):
    file_path: str = Field(default=..., description="Path to the CSV file")


@asset
def cleaned_data(context, config: CleanedDataConfig):
    file_path = config.file_path
    context.log.info(f"Cleaning data from {file_path}")

    data = pd.read_csv(file_path)

    data_cleaned = data.dropna()

    context.log.info("Data cleaning completed.")

    return data_cleaned

etl_job = define_asset_job(name="etl_job", selection=["cleaned_data"], config=RunConfig(
    ops={
        "cleaned_data": CleanedDataConfig()
    }
).to_config_dict())

[garethbrickman]

The value for file_path would have to be typed as a string, so default=... would need to be default="..." if you literally wanted the ellipsis to be the value. Otherwise couldn't you just omit the default parameter entirely?

For making it required, the Field class has an is_required parameter

[eliu-fracta]

In the documentation, that seems a legacy approach? The new way to do it will be using pydantic import Field doc. Addiotionally, I'm using ... instead of "..." to make it a required field based on the documentation.

[garethbrickman]

Ah you're absolutely right, my apologies for getting mixed up with the legacy approach! The way you've set up Field() in your code is valid for the new Pythonic config system.

I think the issue is inside etl_job with how CleanedDataConfig gets instantiated within the RunConfig. Try this format:

etl_job = define_asset_job(name="etl_job", selection=["cleaned_data"], config=RunConfig(
    ops={
        "cleaned_data": {
            "config": {
                "file_path": "path/to/your/csvfile.csv"
            }
        }
    }
).to_config_dict())

[eliu-fracta]

Thanks for answering my question! Is it possible to make the parameter inside RunConfig required as well?

etl_job = define_asset_job(name="etl_job", selection=["cleaned_data"], config=RunConfig(
    ops={
        "cleaned_data": {
            "config": {
                "file_path": "path/to/your/csvfile.csv"  <--- make this required
            }
        }
    }
).to_config_dict())

[garethbrickman]

It will automatically be required in order for the config to be valid for execution otherwise a hard error will occur. Here's how it looks when attempting to materialize within the Dagster UI:

[eliu-fracta]

thank you!