Unmarshal keys containing dots

[inkychris]

Fixes #324

Includes fix #669

• Replaced the use of strings.Split with a method that takes a fully qualified key and returns a list of valid map keys which may include dots in them.

• Reformatted test objects to improve readability and Git change diffing.

• Added key containing dot to yaml test.

[mschneider82]

Hi can you please rebase this into once commit? So we stand a chance to get this reviewed fast and merged?

[inkychris]

Sure thing, will do it when I get home this afternoon.

[albertvaka]

We have very similar code in our fork: https://github.com/DataDog/viper/pull/2/files

[inkychris]

@albertvaka yes looks very similar, although yours uses the keyDelim property which I should change in this.

[xixuejia]

Hello, any update on this? I also encountered this issue

[silvin-lubecki]

Hello @inkychris, any update on this PR? How can I help to make it merge?

[inkychris]

@silvin-lubecki As far as I'm aware, this should be okay to merge; it's just waiting on a maintainer to review/approve.

[spf13]

This looks good. Can you rebase against tip and I'll gladly merge.

[inkychris]

@spf13, this should be good to go now, thanks!

[silvin-lubecki]

ping @spf13 😺

[mschneider82]

please merge @spf13

[inkychris]

@spf13 @sagikazarmark any chance one of you could merge this please? I'd rather not have to keep rebasing... Thanks!

[sagikazarmark]

I'll try to take a look at it soon, but no promises. :)

[mschneider82]

@sagikazarmark ping ;-)

[mschneider82]

Hopefully someone can review this and merge, its so annoying to be stuck on a old forked patched version :( I have settings for "domains", domains always contains dot's...

[sagikazarmark]

I started reviewing this PR. I like the idea and the implementation, but I suspect there might be an edge case where this PR introduces a backwards incompatible change:

emails:
  steve@hacker:
    com:
      created: 01/02/04
      active: false
  steve@hacker.com:
    created: 01/02/03
    active: true

In this case I would expect that emails.steve@hacker.com.active returns false, because that's the original behavior. (Currently the tests fail for the above config, probably for different reasons)

@inkychris can you add a test case (probably with a different email address) that verifies the original behavior remains intact in such cases?

Otherwise 👍 and sorry you had to wait so much for moving forward with this. Thanks @inkychris for the PR.

[inkychris]

For me, the original behaviour results in the value of that key being true rather than false. This is the test I'm running but I could be checking the wrong thing:

package viper

import (
	"github.com/stretchr/testify/assert"
	"strings"
	"testing"
)

const yamlConfig = `
emails:
  steve@hacker:
    com:
      created: 01/02/04
      active: false
  steve@hacker.com:
    created: 01/02/03
    active: true
`

func TestConflict(t *testing.T) {
	SetConfigType("yaml")
	assert.NoError(t, ReadConfig(strings.NewReader(yamlConfig)))
	assert.True(t, GetBool("emails.steve@hacker.com.active"))
}

I should also note I'm testing on Windows with go 1.12 in case this is something that arises from un-ordered parsing or something?

I would expect the issue here is that the address emails.steve@hacker.com.active ambiguously refers to both properties but I guess this is a pre-existing issue anyway?

Apologies, it's been a little while since I thought about this so trying to get up to speed again; thanks for looking at it!

[sagikazarmark]

Hm, I guess you are right. After thinking it through again, it makes sense. Otherwise the fix itself wouldn't work in the first place, since it builds on keys being Getable.

[sagikazarmark]

Thanks @inkychris

[mschneider82]

thanks @sagikazarmark and @inkychris
cannot wait to change my import paths from custom to upstream..

[bogdandrutu]

Any update on this? Can this be merged?

[sagikazarmark]

Yup, I think it's good to go.

[sagikazarmark]

Well, it seems this PR did introduce a regression after all. Dot-notated keys so far were always parsed as structures. Sources, like manually set values, defaults and flags always use dot notated keys, compared to config files:

v.Set("foo.bar", "hello")
v.Default("foo.bar", "hello")
pflag.String("foo.bar", "hello", "")

In this case there is no way to know the original intention (is the key foo.bar or bar under foo?). In the unmarshaling scenario this can probably be solved by including both versions in the source map, but AllSettings should return the same as before.

At this point we either revert this PR, or change it's scope to unmarshal only and generate a map that's compatible with both approaches, although that would potentially be much slower than the current solution.

[bogdandrutu]

In order to support both use-cases (I have an usecase where some keys contain .), if we revert this, can we allow setting the keyDelimiter? I want to set it to .. or something that does allow me to have a key with one ..

[mschneider82]

setting keyDelimiter is fine with me, no solution would be bad

[sagikazarmark]

I submitted a PR reverting the effective changes of this PR (I left the reformatting in place).

Sorry folks, this cannot land in the current version. 🙁

I will send another PR for setting the key delimiter, although we will have to check first if it has any unwanted effects.

[inkychris]

The underlying issue was conceptual (not being able to determine the key path components from it's fully qualified path) which is only really solvable by either escaping or changing the delimiter used, so it's understandable.

Just a heads up for the following PR that the following line appears to have missed the keyDelim refactoring, in case tests don't hit this:

viper/viper.go

Line 1441 in 99520c8

path := strings.Split(key, ".")

[sagikazarmark]

Thanks for the heads up @inkychris !