New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[user config] exception with nested structures in default context #1149
Comments
Fixed KeyError exception when YAML user config file contains nested structures by settting the default's default to an OrderedDict if the key does not exist. Added test_get_config:test_get_config_with_nested_struct_defaults to cover that case
This bug seems to be due to how # Possible solution 1: Use dict.update instead of recursing.
if isinstance(v, dict):
new_config[k].update(v)
# Possible solution 2: Only recurse for known top-level keys.
if isinstance(v, dict) and k in DEFAULT_CONFIG:
new_config[k] = merge_configs(default[k], v) This issue is likely also related to #1232. |
It's been a while ... so I had to refresh my memory. I think solution 1 wouldn't work because if the user config is a subset of the entire config (why repeat config options we want to keep as default) then any nested dictionary would override whatever dictionary is at that level e.g. default = dict(a=1, b=2, c=dict(c1=3, c2=4))
user = dict(a=1, b=4, c=dict(c1=5))
default.update(user)
# Resulting in {'a': 1, 'b': 4, 'c': {'c1': 5}} As far as solution 2, the recursion would only happen if k is in DEFAULT_CONFIG, which, unless I am not understanding something, would not solve the problem. Here is how I had fixed the problem. Not sure if this is the most elegant way to solve it but it does work. def merge_configs(default, overwrite):
"""Recursively update a dict with the key/value pair of another.
Dict values that are dictionaries themselves will be updated, whilst
preserving existing keys.
"""
new_config = copy.deepcopy(default)
for k, v in overwrite.items():
# Make sure to preserve existing items in
# nested dicts, for example `abbreviations`
if k not in overwrite:
raise KeyError("option '{}' is not defined in configuration {}".format(k, overwrite))
if isinstance(v, dict):
new_config[k] = merge_configs(default.setdefault(k, collections.OrderedDict([])), v)
elif isinstance(overwrite[k], list) is True and isinstance(v, list) is False:
# The default option is a list. If the value is a scalar, we need to make sure
# that v is in the list
if v not in overwrite[k]:
raise ValueError("value '{}' of option '{}' "
"is not in the list of "
"choices {}".format(v, k, overwrite[k]))
new_config[k] = v
else:
new_config[k] = v
return new_config |
I am hit by the same issue. "algorithms": {
"dummy" : {
"id" : "dummy_algo",
"roles": ["part1", "part2"]
} and I want to be able to replace the content of algorithms:
algo1:
id: "myalgo1"
roles:
- part1
- part2
algo2:
id: "myalgo2"
roles:
- part1 and the solution proposed by @lbrack fixes my issue. |
@ssbarnea you removed this from 1.8.0 could you elaborate why? Without this user-overriding of dict variables is not possible. in the light of this another solution would be to treat those dict var as: extra_context: in the yaml config file. |
I removed the label and assigned the milestone because labels should not be used with version planning for which we already have milestones (or projects). |
ok thanks for the workflow explanation @ssbarnea |
For those who need a workaround today, you can just overwrite the cookiecutter.json file with the values you want. and run cookiecutter --no-input <path/to/cookiecutter> |
I applied the fix from @simobasso and it worked. |
just merged #1516 🎉 |
Well but with cookiecutter version 1.7.3 this still does not work... |
@paul-michalik changes of #1516 were not yet released. |
When do you plan to release it? Is it scheduled for specific version? |
It should be released with |
Description:
After reading the section on user config and dictionary variables I was under the impression that defining nested YAML data structures under default_context would work (but it fails with a key error exception as is it attempting to access a key that doesn't exist in DEFAULT_CONFIG when iterating recursively through the dictionary resulting from the user config).
After looking at the code the root cause is pretty obvious (and so is the fix) but I am wondering if this is intended behavior (still it shouldn't fail with an exception).
Just for the sake of clarity here is a minimalist user config that makes cookiecutter fail
What I've run:
To reproduce this, copy the YAML example given above in user_config.yml and run
cookiecutter https://github.com/audreyr/cookiecutter-pypackage.git --config-file=user_config.yml
The actual template is irrelevant since the failure occurs before it is even fetched. Here is the backtrace
Version
The text was updated successfully, but these errors were encountered: