Nested env #3159

Air-Mark · 2021-09-01T17:47:24Z

Change Summary

new env_nested_delimiter option in Config class
optional _env_nested_delimiter kwarg in init BaseSettings
patched EnvSettingsSource
Depending on env_nested_delimiter trying to find env variables with that delimiter and split those variable into nested dicts. Merge those dicts if they have same nesting structure.
So lets say we have

class A(BaseSettings):
    foo: str


class B(BaseSettings):
    bar: A


class C(BaseSettings):
    baz: B

now we are able to fill the A.foo with env variable like
BAZ__BAR__FOO='hello'

Related issue number

#2304 #2232

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)
My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

MadJlzz · 2021-09-22T12:48:36Z

Exactly what I was looking for! I believe that pydantic is already doing an excellent job ; removing boiler plate code thanks to this PR will help us centralize settings configuration.

I can help if needed (doc, testing, ...)

cc @daviskirk, because https://github.com/daviskirk/climatecontrol#pydantic seems to have this already implemented.

# Conflicts: # pydantic/env_settings.py

Air-Mark · 2021-09-25T14:48:34Z

@MadJlzz it is nice to hear that it was useful for you. Sorry for disappearing for almost a moth... I cleaned up code and logic a bit. Added some docs and tests. I think this PR is now ready for submission. And of course It would be great to have some help if needed.

Air-Mark · 2021-09-25T14:57:32Z

please review

samuelcolvin

Thanks for this, I've wanted this since I first build pydantic, I didn't include it in the first release and never got around to implementing it.

I'm really excited to use this once it's done.

docs/usage/settings.md

pydantic/env_settings.py

samuelcolvin · 2021-12-08T21:15:24Z

please update

Co-authored-by: Samuel Colvin <samcolvin@gmail.com>

Air-Mark · 2021-12-11T19:15:06Z

please review

samuelcolvin

A few things to change, you also need to rethink explode_env_vars significantly.

pydantic/env_settings.py

samuelcolvin · 2021-12-12T09:58:31Z

docs/examples/settings_nested_env.py

+os.environ.update({
+    'top': '{"v1": "1", "v2": "2"}',
+    'v0': '0',
+    'top__v3': '3',
+    'top__sub': '{"sub_sub": {"v6": "6"}}',
+    'top__sub__v4': '4',
+    'top__sub__v5': '5',
+})


as mentioned before, please move this to exec_examples.py

Fixed. Misunderstood you last time about where exactly I should place this.

samuelcolvin · 2021-12-12T10:05:06Z

pydantic/env_settings.py

+            for idx, key in enumerate(keys):
+                if idx == len(keys) - 1:
+                    try:
+                        env_val = settings.__config__.json_loads(env_val)  # type: ignore


I don't understand why you need this here? Also not sure why we're catching and ignoring TypeError

Agreed about the TypeError we process only env values here, so there are only sting values.
We need this piece because env_val can be a json string. And there could be a situation when you want to mix both approaches of defining nested models:

export TOP__SUB__V5='5' export TOP='{"sub": {"v6": "6"}}'

So the correct behaviour in my mind was that we should merge those variables into one, instead of overwriting.

samuelcolvin · 2021-12-12T10:05:51Z

pydantic/env_settings.py

+        result: Dict[str, Any] = {}
+        for env_name, env_val in env_vars.items():
+            keys = env_name.split(self.env_nested_delimiter)
+            env_var = result


I don't think this approach works.

If you change the order of env variables in your test below, the result changes. E.g. put env.set('top__sub__v5', '5') first.

This was the right logic, because this variable should be overwritten by the next one env.set('top', '{"v1": "1", "v2": "2", "sub": {"v5": "xx"}}')
But thank you for pointing on this. Double checked all combinations and yes there was a bug. Redesigned this method and test a bit according to this bug and logic from my answer on your comment above.

samuelcolvin · 2021-12-12T10:07:15Z

please update.

Air-Mark · 2021-12-12T19:30:41Z

Thank you for review. Did changes that you been asking for. Improved and redesigned explode_env_vars a bit.
Please review

samuelcolvin · 2021-12-18T20:24:27Z

Thanks so much for this, I'm really looking forward to using it.

I've made some changes to the logic to avoid a few errors I found. Please let me know if you have any and feedback or want to make any more changes, otherwie I'll merge over the next few days.

samuelcolvin · 2021-12-18T20:57:17Z

Thanks so much for this, I've actually merged now as I want to include this in v1.9 and I want to get a pre-release version of v1.9 out today.

If you want to make any changes to this, feel free to create new PR.

Air-Mark · 2021-12-19T05:17:53Z

That is awesome. Thank you for review and help. I can use a pre-release version in some projects, and will be glad to help with bugs if needed.

Mark Trifonov added 3 commits September 1, 2021 17:59

Environment names for complex types pydantic#2304

46eea4b

nested env disabled by default

730c039

cleanup

3518f5e

Mark Trifonov added 5 commits September 25, 2021 13:27

Merge branch 'master' into nested_env

b9ed167

# Conflicts: # pydantic/env_settings.py

nested env settings: simplified and mypy fixes

6c457ce

nested env settings: config, test, doc

ae9592e

nested env settings: changes file

88aae14

nested env settings: cleanup

1994a92

Air-Mark marked this pull request as ready for review September 25, 2021 14:48

github-actions bot added the ready for review label Sep 25, 2021

github-actions bot assigned PrettyWood and samuelcolvin Sep 25, 2021

samuelcolvin reviewed Dec 8, 2021

View reviewed changes

github-actions bot added awaiting author revision and removed ready for review labels Dec 8, 2021

github-actions bot assigned Air-Mark and unassigned samuelcolvin and PrettyWood Dec 8, 2021

samuelcolvin mentioned this pull request Dec 8, 2021

feat: add support multiple dotenv files (#1497) #3222

Merged

5 tasks

Air-Mark and others added 2 commits December 11, 2021 22:11

Apply suggestions from code review

581906b

Co-authored-by: Samuel Colvin <samcolvin@gmail.com>

Apply suggested changes from code review

0ab431a

github-actions bot added ready for review and removed awaiting author revision labels Dec 11, 2021

github-actions bot assigned PrettyWood and samuelcolvin Dec 11, 2021

github-actions bot unassigned Air-Mark Dec 11, 2021

lint fix

63ffab3

samuelcolvin reviewed Dec 12, 2021

View reviewed changes

github-actions bot added awaiting author revision and removed ready for review labels Dec 12, 2021

github-actions bot assigned Air-Mark and unassigned samuelcolvin and PrettyWood Dec 12, 2021

changes from code review

4a83110

github-actions bot added ready for review and removed awaiting author revision labels Dec 12, 2021

github-actions bot assigned PrettyWood and samuelcolvin and unassigned Air-Mark Dec 12, 2021

samuelcolvin added 3 commits December 18, 2021 20:07

simplify explosing env vars

67d5680

Merge branch 'master' into Air-Mark-nested_env

c97138a

linting

5e9b229

samuelcolvin merged commit be24670 into pydantic:master Dec 18, 2021

Air-Mark mentioned this pull request Jan 7, 2022

fix ability to mix/merge json env variables with env_nested_delimiter… #3640

Closed

5 tasks

earmenda mentioned this pull request May 18, 2022

Add a credentials config which supports custom keys ethyca/fides#673

Closed

16 tasks

hramezani mentioned this pull request Jan 22, 2024

Add CLI Settings Source pydantic/pydantic-settings#214

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nested env #3159

Nested env #3159

Air-Mark commented Sep 1, 2021 •

edited

MadJlzz commented Sep 22, 2021

Air-Mark commented Sep 25, 2021

Air-Mark commented Sep 25, 2021

samuelcolvin left a comment

samuelcolvin commented Dec 8, 2021

Air-Mark commented Dec 11, 2021

samuelcolvin left a comment

samuelcolvin Dec 12, 2021

Air-Mark Dec 12, 2021 •

edited

samuelcolvin Dec 12, 2021

Air-Mark Dec 12, 2021

samuelcolvin Dec 12, 2021

Air-Mark Dec 12, 2021 •

edited

samuelcolvin commented Dec 12, 2021

Air-Mark commented Dec 12, 2021

samuelcolvin commented Dec 18, 2021

samuelcolvin commented Dec 18, 2021

Air-Mark commented Dec 19, 2021

Nested env #3159

Nested env #3159

Conversation

Air-Mark commented Sep 1, 2021 • edited

Change Summary

Related issue number

Checklist

MadJlzz commented Sep 22, 2021

Air-Mark commented Sep 25, 2021

Air-Mark commented Sep 25, 2021

samuelcolvin left a comment

Choose a reason for hiding this comment

samuelcolvin commented Dec 8, 2021

Air-Mark commented Dec 11, 2021

samuelcolvin left a comment

Choose a reason for hiding this comment

samuelcolvin Dec 12, 2021

Choose a reason for hiding this comment

Air-Mark Dec 12, 2021 • edited

Choose a reason for hiding this comment

samuelcolvin Dec 12, 2021

Choose a reason for hiding this comment

Air-Mark Dec 12, 2021

Choose a reason for hiding this comment

samuelcolvin Dec 12, 2021

Choose a reason for hiding this comment

Air-Mark Dec 12, 2021 • edited

Choose a reason for hiding this comment

samuelcolvin commented Dec 12, 2021

Air-Mark commented Dec 12, 2021

samuelcolvin commented Dec 18, 2021

samuelcolvin commented Dec 18, 2021

Air-Mark commented Dec 19, 2021

Air-Mark commented Sep 1, 2021 •

edited

Air-Mark Dec 12, 2021 •

edited

Air-Mark Dec 12, 2021 •

edited