[bug] Using environments with dynaconf_merge causes duplicate list items

[ZianVW]

Describe the bug
When I use the layered environments functionality with dynaconf_merge=True, I unexpectedly get repeated list items.

To Reproduce
Steps to reproduce the behavior:

1. Having the following folder structure

Folder structure does not matter. Attached sample code places all files in a single folder.

dynaconf.zip

Project structure

.env
.gitignore
.secrets.yaml
config.py
main.py
requirements.in
requirements.txt
settings.yaml

2. Having the following config files:

Config files

/path/.env

ENV_FOR_DYNACONF=env1

and

/path/settings.yaml

---
dynaconf_merge: True
default:
    my_list:
        - default_item
env1:
    my_list:
        - env1_item1
        - env1_item2

3. Having the following app code:

Code

/path/src/main.py

from config import settings
from pprint import pp

pp(settings.as_dict())

4. Executing under the following environment

Execution

# other commands and details?
# Activated a clean virtualenv with only dynaconf 3.1.7 installed.

$ python /path/src/main.py
{'LOAD_DOTENV': True,
 'MY_LIST': ['default_item',
             'env1_item1',
             'env1_item2',
             'env1_item1',                          <---- See that these list items appear twice
             'env1_item2']}

Expected behavior
I expect the following result in settings:

{'LOAD_DOTENV': True,
 'MY_LIST': ['default_item',
             'env1_item1',
             'env1_item2']}

Environment (please complete the following information):

• OS: Windows 10 Enterprise / 21H1
• Dynaconf Version: 3.1.7
• Frameworks in use: None

Additional context
Add any other context about the problem here.

[rochacbruno]

#661 and #663 fixed this issue

Have you tried with latest master?

pip install https://github.com/rochacbruno/dynaconf/tarball/master

[ZianVW]

Moving the dynaconf_merge: True statement from the file level to the environment level also causes an issue.

settings.yaml:

---
default:
    my_list:
        - default_item
env1:
    dynaconf_merge: True
    my_list:
        - env1_item1
        - env1_item2

Output:

python.exe main.py
{'LOAD_DOTENV': True,
 'MY_LIST': ['default_item',
             'env1_item1',
             'env1_item2',
             'env1_item1',
             'env1_item2']}

Process finished with exit code 0```

[rochacbruno]

I will add your examples as test cases together with existing tests on https://github.com/rochacbruno/dynaconf/blob/master/tests/test_yaml_loader.py#L428-L493

However I think your first case is the intended behavior, there are cases when user wants to have items duplicated.

[shvilime]

I have the same problem with merging list of sting with last version of dynaconf 3.1.7
1.toml

dynaconf_merge = true
[development]
mas = ["1"]

2.toml

dynaconf_merge = true
[development]
mas = ["2"]

config.toml

[development]
dynaconf_include = ["1.toml", "2.toml"]

dyn.py

from dynaconf import Dynaconf
print(Dynaconf(settings_file="config.toml", environments=True, env="development").mas)
print(Dynaconf(settings_file="config.toml", environments=True).mas)

The result is:
["1", "2", "2"]
["1", "2"]

@rochacbruno is it normal behavior ?

[andressadotpy]

Hello @ZianVW, I just added a test to reproduce your error, but I'm not being able to, it's passing that way:

def test_list_entries_from_yaml_should_not_duplicate_when_merge_True_mark_on_file(tmpdir):
    data = {
        "default": {
            "my_list" : ["default_item"]
        },
        "other": {"dynaconf_merge": True, "my_list":["other1", "other2"]},
    }
    yaml_loader.write(str(tmpdir.join("test_settings.yaml")), data)

    settings = Dynaconf(
        settings_files="test_settings.yaml",
        environments=True,
        merge_enabled=True,
    )

    expected_value = BoxList(["default_item", "other1", "other2"])

    assert settings.from_env("other").MY_LIST == expected_value

Does this test seems to be doing the same thing that you are trying to do?

[andressadotpy]

For the above test, when I add an assertion:

assert settings.MY_LIST == expected_value

the value of MY_LIST isn't updated with the others, is this the expected behavior @rochacbruno?

E   AssertionError: assert <BoxList: ['default_item']> == <BoxList: ['d...1', 'other2']>
E     Right contains 2 more items, first extra item: 'other1'
E     Full diff:
E     - <BoxList: ['default_item', 'other1', 'other2']>
E     + <BoxList: ['default_item']>

[rochacbruno]

I think this is fixed by #810