Utility function to read snakemake rule

[Manangka]

In GitLab by @Huite on Aug 14, 2023, 11:51

To run workflows both interactively and in a snakemake workflow, this does the trick:

def read_snakemake_rule(path, rule: str) -> "snakemake.rules.Rule":
    """
    Parameters
    ----------
    path: str, pathlib.Path
        The path to the snakefile.
    rule: str
        Name of the rule in the snakefile that runs this script.
        
    Returns
    -------
    rule: snakemake.rules.Rules
    
    Examples
    --------
    To run an example both interactively and in a workflow, e.g.:
    
    >>> if "snakemake" not in globals():
    >>>     snakemake = read_snakemake_rule("snakefile", rule="my_rule")
    >>> modelname = snakemake.params.modelname
    >>> template = snakemake.input["template"]
    """
    import snakemake as sm
    
    workflow = sm.Workflow(snakefile="snakefile")
    workflow.include(path)
    return workflow.get_rule(rule)
    

It's probably a good idea to add this to imod.util.

[JoerivanEngelen]

I tried this and it appears the Snakemake API has changed and this doesn't seem to work anymore from a python session. There is some internal documentation on the API, but it is very minimal: https://snakemake-api.readthedocs.io/en/stable/api_reference/snakemake_api.html
It's stated that this is for Snakemake internal developers only. The furthest I got with the most recent API is this:

def read_snakemake_workflow(path) -> "snakemake.rules.Rule":
    """
    Parameters
    ----------
    path: str, pathlib.Path
        The path to the snakefile.
    rule: str
        Name of the rule in the snakefile that runs this script.
        
    Returns
    -------
    rule: snakemake.rules.Rules
    
    Examples
    --------
    To run an example both interactively and in a workflow, e.g.:
    
    >>> if "snakemake" not in globals():
    >>>     snakemake = read_snakemake_rule("snakefile", rule="my_rule")
    >>> modelname = snakemake.params.modelname
    >>> template = snakemake.input["template"]
    """
    import snakemake.settings as settings
    from snakemake.api import SnakemakeApi
    
    with SnakemakeApi(
        settings.OutputSettings(
            verbose=True,
            show_failed_logs=True,
        ),
    ) as snakemake_api:
        workflow = snakemake_api.workflow(
            storage_settings=settings.StorageSettings(),
            resource_settings=settings.ResourceSettings(),
            snakefile=Path(path),
        )

    # get_rule method disappeared
    return workflow

# This prints rules as type `odict_values`, but the original ordinary dictionary I could not find
print(workflow._workflow.rules)

The docs refer to a --debug option in the CLI, but how to use this I can't seem to find anywhere. It didn't seem to do anything when I tried it.

I think the Snakemake API is subject to changes, and therefore keeping a utility like this alive is quite maintenance costly.
I'm closing this for now, if a good alternative pops up, we can reopen this issue again.

[Huite]

I think the Snakemake API is subject to changes, and therefore keeping a utility like this alive is quite maintenance costly.

Keep in mind what the alternative is here: multiple people have already come up with tricks to parse snakefiles to extract values out of them.
I agree that this is tedious to maintain: IIRC, I pieced the above implementation together from a snakemake issue.

The trade-off is here: either we do it with an hour of maintenance here and there, or there will be multiple people spending over a day of their time on this!

[JoerivanEngelen]

I think an hour of maintenance here and there is an underestimation of the amount of work required.
FYI: I spent 4 hours on this without coming to a satisfactory replacement of the proposed function for Snakemake 8.
I just found that the announcement for the changes was in this issue here:
snakemake/snakemake#2409

Breaking changes upon a snakemake release can stall the development team with a day easily fixing a broken pipeline, as documentation of its internals is (currently?) insufficient. Furthermore, it adds an extra layer of complexity for developers of iMOD Python to understand. Finally, Snakemake would be quite a heavy (optional) extra dependency, though the developers work on making it less heavy with a conda injection system.

I'd rather create a separate package for this feature, to separate concerns. This could also be used by other colleagues at Deltares who do not require the other features of iMOD Python. I found a similar initiative to this, but unfortunately it seems stale now for 1 year, thus not compatible with Snakemake 8.
https://github.com/teristam/snakehelper

[Huite]

Separate package is indeed best, because hydromt users would also benefit.

And we can indeed isolate it so it doesn't block CI.

Ideally though, this would just work through snakemake right? I searched through the snakemake issues back then, but I don't remember my conclusion.

[Huite]

Another thought: maybe add the snippet above to the FAQ with a note that it works on snakemake 8, but we don't guarantee it working always.