[RFC] Config provider mechanism

[sagikazarmark]

Intro

Viper's main advantage over similar solutions is that it's highly integrated and provides tons of features out of the box. The relatively flat API (no complex wiring is necessary to make it work) is appealing to many people. However, it became cluttered over the years and new features were added to it. Similarly, the architecture behind Viper became quite complex as a result of a flat design and continuous development which makes maintenance (and accepting PRs in particular) a lot harder, especially if we consider that Viper is a central component to a lot of applications.

In Viper v2 we have a chance to make large scale changes that'll make the project more maintainable and better on the long term.

In #886 I've explained how I imagine separating certain functionalities of Viper, improving the overall architecture. In this issue, I'd like to expand on a potential config provider mechanism (or abstraction if you like).

Goals and non-goals

The goal of this proposal is to define a low level configuration loading API. It should be built largely based on the experience gathered during the maintenance of Viper. The idea is to construct an API that supports every (or most) configuration loading use case supported by Viper at the moment.

The resulting API does not have to provide the same user experience though. The goal is to come up with a consistent API that works for all config sources.

It also does not have to be perfect on the first attempt as it will remain internal until we are satisfied with the result.

I'm a huge fan of the Go 1 BC promise and that features are implemented in a backwards compatible way when possible and I think we should follow that mentality in Viper, so ideally it should be possible to reintegrate this new API into Viper v1. Although I don't consider that a requirement, I also wouldn't consider it a success should we fail to do that.

Previous attempt

#986 is a minor refactor of the encoding layer. Minor in terms of LOC, but a major change in Viper's architecture. It grabs a piece of common functionality, extracts it to a separate module (or modules in this case) and reintegrates it to Viper (see my comment about BC in the Goals and non-goals section).

Although the final version of the PR focuses on encoding/decoding, there are two things related to configuration loading in it:

• In a previous version, it supported "files" (readers). Considering that data retrieved from remote key-value stores also gets "decoded", it was kind of an early implementation of a "file source". The PR was repurposed when I realized that the abstraction was kind of flawed that way.
• Although the interfaces in the PR do not return or accept map[string]interface{} internally some implementations require special handling for that type. A future version of these interfaces will probably target map[string]interface{} directly.

Prior art

There are various configuration libraries out there, but the most prominent one (other than Viper) is probably go-micro's config. There are a couple major differences between Viper and go-micro config (let's just call it go-micro going forward):

• go-micro already has a layered architecture for config loading and encoding whereas Viper has a strong focus on a flat API, optimizing for user experience.
• go-micro has a single interface for config sourcing, including watching configuration out of the box for each source (more about this later).
• go-micro deals with encoded data, meaning that even flags and environment variables are encoded to some raw format (most probably to JSON by default). Internally, it uses map[string]interface{} to merge configuration from different sources. Viper uses map[string]interface{} on all levels.
• go-micro made some opinionated decisions about how configuration keys are parsed (particularly in case of flags and environment variables)

Architecturally speaking, go-micro is an excellent library and it serves as a great example of what we should aiming for in Viper (with the low level API anyway), but some of the design decisions don't really fit into the Viper ecosystem, namely:

• data representation
• opinionated key parsing

When looking at go-micro as an example, we should keep these in mind to maintain what makes Viper great.

API

With those in mind, let's take a look at the proposed API. This isn't the first draft I come up with, but probably isn't the last either. Also, the terminology isn't rock solid yet.

Config source

A config source can be any kind of data source that provides data that can be translated to map[string]interface{}.

// Config source is a data source that provides data translateable to configuration.
type ConfigSource interface {
	Values() map[string]interface{}
}

Viper (or any higher level configuration loader entity) should be able to call Values any time at their discretion.

Config source loader/initializer

The ConfigSource interface doesn't address the fact that some config sources might be stateful and that an internal state must be loaded before using it. Flags and files are great example: command arguments and files should not be loaded again when values are accessed.

Therefore we need an optional interface that a ConfigSource instance can implement:

// ConfigSourceLoader is an optional interface for a ConfigSource.
// It allows loading an internal state before accessing it.
// If a ConfigSource implements it, it MUST be called exactly once before calling Values.
type ConfigSourceLoader interface {
	Load() error
}

Alternatively, we can think of it as initialization (not all initialization steps might actually load state, for example connecting to remote sources could be considered initialization):

// ConfigSourceInitializer is an optional interface for a ConfigSource.
// It allows executing initialization steps before accessing its internal state
// (for example: constructing remote clients, loading files, parsing flags, etc).
// If a ConfigSource implements it, it MUST be called exactly once before calling Values.
type ConfigSourceInitializer interface {
	Init() error
}

Making a ConfigSource stateful begs the question whether Values should be aware of initialization:

• If not then it will simply return an empty map for an uninitialized source
• If it should be, we have two options:
  • Values should initialize the source
  • Values should NOT initialize the source

If Values should be aware of the state, we need to change its signature:

// ConfigSource is a data source that provides data translateable to configuration.
type ConfigSource interface {
	Values() (map[string]interface{}, error)
}

This signature is also better for remote config providers.

Config source defaults

Certain config sources (eg. flags) can provide their own defaults. Currently we manually check flag defaults if a certain key does not have a value in Viper. Config sources could implement an interface that tells Viper that the source allows defining defaults and return them, similarly to Values:

// ConfigSourceDefaulter is an optional interface for a ConfigSource.
// It tells a higher level API that this particular ConfigSource allows defining default values.
type ConfigSourceDefaulter interface {
	Defaults() map[string]interface{}
}

Config source watchers

Config sources like files and remote key-value stores often allow watching for configuration changes. There are several different options we can choose from. One is registering a callback in a watcher:

// ConfigSourceWatcher is an optional interface for a ConfigSource.
// If a ConfigSource implements it, it MUST watch the configuration for changes (eg. through file watchers)
// and call the callback function when that happens.
type ConfigSourceWatcher interface {
	Watch(func(map[string]interface{}))
}

This has it's downsides though: there is less room for error handling.

Another alternative is a custom watcher interface:

// ConfigSourceWatcher is an optional interface for a ConfigSource.
// If a ConfigSource implements it, it MUST return a ConfigWatcher
// that can be used to subscribe for configuration changes.
type ConfigSourceWatcher interface {
	Watch() (ConfigWatcher, error)
}

// ConfigWatcher watches a configuration source for changes and returns the new configuration
// when a change is detected.
// Values should block until a new change is detected.
// Stop should stop the watcher.
type ConfigWatcher interface {
	Values() (map[string]interface{}, error)
	Stop() error
}

This provides more options for error handling and gives better control to the caller.

Summary

This proposal tries to cover most of the config sourcing uses cases currently available in Viper. The idea is to factor out these implementations to better tested components and reintegrate them to work in the current Viper API. There is a chance that some things are missing or that the abstraction is not perfect (or too complex), but the plan is to implement config sources in internal packages and improve them until we are satisfied with the result.