Design Discussion: Map Storage Providers with multiple configurations

[sguilhen]

This discussion aims to address the issue repored in #10226

The problem

To summarize the issue, we can have areas, such as role, group, user, etc in the system configuration that use the same MapStorageProvider implementation - e.g. jpa-map-storage, but with different configurations. Currently the storage providers can define a root configuration in the mapStorage SPI, and areas that need to use a different storage configuration have to specify the storage configuration as part of their own config. Something like this:

    "mapStorage": {
            ...     
            "jpa-map-storage": {
                # root db configuration - e.g. postgres running on hostA
            }
    },
    
    "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage"
                # area that uses the root jpa-map-storage configuration
            }
        }
    },

    "client": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "jpa-map-storage": {
                        # area needs a different configuration - e.g. this would work with a cockroach database
                }
            }
        }
    }

However, all the storage factories for the different areas are assigned the same componentId, whose format is mapStorage-f:ROOT:storage_name and the first storage factory that is instantiated is cached under the componentId key, which in the end makes all areas use the same configuration. So instead of having role and client using different DBs they both end up in the same DB - the one specified in the configuration of the area that is requested first.

So the issue is that the componentId does not consider the configuration of the storage provider, and we need to think about ways to fix this.

Possible Solutions

Include the configuration hashcode in the componentId
Two configs for a storage provider are equal if they yield the same values for the set of supported configuration attributes. This means that we could add a getSupportedAttributes method to the factories (perhaps to the AmphibianProviderFactory) that returns the set of attributes the particular implementation supports, and then we could fetch the attribute values using the Scope that is constructed for the particular area storage and store all key/value pair in a HashMap. The final map's hashcode can then be appended to the componentId.

With this approach, two different areas that have the same config end up sharing the same factory. However, this means that every time a storage is requested we will have to compute this hashcode to determine if we can reuse a previously instantiated factory. In addition, if two or more areas need to share the same alternative configuration, we still need to replicate the config for each area.

Configure the componentId manually for each area
In this case, one could manually set the componentId for the areas, and those with the same componentId end up sharing the same storage factory. Something like this:

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "componentId": "some-unique-string",
                "jpa-map-storage: {
                        # alternative configuration
                }
            }
        }
    },

    "client": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "componentId": "some-unique-string",
                "jpa-map-storage": {
                        # alternative configuration
                }
            }
        }
    }

This would work but it we still need to replicate the config for all areas that want to share an alternative configuration because we don't know which area will be requested first (and thus have its storage factory initialized and cached for the others to use).

Proposal: add profiles for the storage factory configurations and have areas reference specific profiles

The idea here is to expand the configuration of the storage providers to support multiple profiles, and have each area reference one of the profiles in their own configuration. The componentId would be computed as mapStorage:storage_provider:profile. For example, if we have a profile named postgres-dev for the jpa-map-storage provider, the componentId would be computed as mapStorage:jpa-map-storage:postgres-dev.

The configuration of a root storage factory would look like this (NOTE: in the examples we're using the jpa-map-storage but the idea can be applied to any storage provider):

    "mapStorage": {
        ...     
        "jpa-map-storage": {
            "default-profile": "postgres-dev",
            "profile": {
                "postgres-dev": {
                    # configuration of the postgres-dev database.
                },
                "postgres-prod": {
                    # configuration of the postgres production database.
                }
            }
        }
    },

default-profile is optional, in the absence of this attribute we could look for a profile named default and if there is no such profile we could use the factory's root configuration as the scope that is fed to init. For example,

    "jpa-map-storage":  {
          # some db configuration, no profile
    }

Would be equivalent of a configuration that has a single profile named default. i.e.

    "jpa-map-storage":  {
        "profile" : {
            "default": {
                # some db configuration
            }
        }
    }

Now each individual area would be able to specify the profile in its configuration and reuse the same config. For example,

 "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "provider-profile": "postgres-dev"
            }
        }
    }

Would use the postgres-dev profile of the jpa-map-storage provider for the roles. In the absence of the provider-profile, the storage factory's default profile would be used (and for that we would check what default was configured in the root provider configuration). That is

 "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
            }
        }
    }

Would end up using the default profile for the jpa-map-storage. If our config looks like the first example in this section, we would end up using the postgres-dev profile.

Every area using the same storage provider and profile would end up having the same componentId and thus share the same storage provider factory. In addition, this no longer requires copying and pasting the same config for every area that needs to use a different database configuration. Finally, this solution makes it easier to switch between dev and production profiles by having all areas use the default profile and then change the default to a different profile in the root configuration.

Area-specific storage provider configuration

It would still be possible to have a config specific to an area. One way to do it would be to define a separate profile that is used only by that single area. A different approach would be to have the storage config specified in the area itself, much like what we do today:

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "jpa-map-storage: {
                        # alternative configuration
                }
            }
        }
    }

In this case, we check if there is an inline configuration for the storage and if we have one, we can generate a componentId just for the specific area. Something like mapStorage:jpa-map-storage:role-inline. As a result it would have its own storage factory.

The inline config would still fallback to the root configuration as we have today. In other words, if an attribute can't be found in the inline config it checks the root config. The only difference is that in this case we would fallback to one of the profiles. If the provider-profile isn't specified, we fallback to the default profile as determined by the root configuration. If the provider-profile is specified, we fallback to the specific profile.

So, the example above would fallback to the postgres-dev profile as that is the default profile in the root configuration. So the role config can, for example, have just a different connection URI that uses a different DB, and get all other attributes, such as user, password, etc, from the postgres-dev profile.

The following inline config

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "provider-profile": "postgres-prod"
                "jpa-map-storage: {
                        # alternative configuration
                }
            }
        }
    }

Would use the inline config and fallback to the postgres-prod profile.

Of course there are a few implementation-specific details that would need to be sorted out to make all this work, but it is flexible in the sense that it expands the storage factories configuration and it allows for the areas to use different storage configurations simply by referencing them instead of copying and pasting the config into each area. Plus, it can still provide a way for an area to have a very specific config by keeping the inline configuration that falls back into one of the profiles.

Thoughts, comments and other considerations are highly welcome!

[sguilhen]

Another upside of the profile-based approach is that we could even pre-load all the storage factories, instantiating one for each profile and then computing the componentId and caching them in the DefaultComponentFactoryProviderFactory before they are even requested by an area.

[ahus1]

Thanks for proposing this change. Once mapStorage gets traction, I see that there will be multiple instances of a variant like JPA or LDAP most of the time.

My comments on this:

I like the solution where a configuration is stated centrally in mapStorage and is then referenced in other places as it prevents duplicating the configuration by an ID. To name this ID a profile is fine for me, or it could be just named "id".
I could even see that a storage must be configured in mapStorage and can't be configured locally to avoid having too many different variants. Also I'd rather skip the defaultProfile for an always explicitly mentioned profile.

One thing I didn't see in the examples you provided: How would I configure a store per realm? I can image that users want to map a realm to a database, or in LDAP scenarios would map a realm to a specific LDAP directory or subtree. While this question is quite important for me from an LDAP point of view, feel free to explain if this is not related to this question.

[sguilhen]

Thanks for the comments, @ahus1! For the moment I will stick to having a default profile just to avoid enforcing users to explicitly configure a profile in cases where they only have one possible config at the provider root config, but this is of course subject to changes.

The problem with the configuration of the providers as components in a realm is one that is still not solved and is still a work in progress. There are a few points that need to be addressed on that front, so as of now we simply don't know how exactly that will look like.

[hmlnarik]

The realm configuration will be done via declaring a component for map storage / map provider (to be decided) inside that realm, and then requesting a map storage using a component lookup prior to using a system-wide one.

[sguilhen]

I've crafted a draft PR with the changes proposed here - #10429. Testing locally I was able to have some models (role) in one DB, and others (client, clientScope) in a different DB. It needs some tidying up but it gives an idea of what would be needed to make it work.

[hmlnarik]

Thank you @sguilhen for the proposal. I very much like the idea of ability to declare profiles within a system configuration.

I would like to stress the scope of this proposal: The proposal is limited for use in system configuration; configuration profile declaration is not suitable for realm components (since component configuration is not a tree but a map of configuration options, and profile declaration effectively builds a tree structure).

There are few comments / clarifications needed, and I will add those as separate answers to this discussion so that they can be discussed independently.

[sguilhen]

I would like to stress the scope of this proposal: The proposal is limited for use in system configuration; configuration profile declaration is not suitable for realm components (since component configuration is not a tree but a map of configuration options, and profile declaration effectively builds a tree structure).

Definitely, thanks for stressing this. The proposal's scope is indeed limited to the system configuration.

[hmlnarik]

Default configuration

The default profile configuration should be indistinguishable from the same configuration of provider when there is no profile defined, i.e. the following two configs should be equivalent for achieving "postgres-dev" configuration:

    "mapStorage": {
        ...     
        "jpa-map-storage": {
            "driver": "XYZ",
            "datasource": "FOO"
             # configuration of the postgres-dev database.
    } },

and

    "mapStorage": {
        ...     
        "jpa-map-storage": {
            "driver": "XYZ",
            "datasource": "FOO",
            // configuration same as of the postgres-dev profile.
            "profile": {
                "postgres-dev": {
                    // "datasource": "FOO" <-- this has been here originally and should not have been, moved above
                    // this profile configuration is intentionally left empty
                },
                "postgres-prod": {
                    "datasource": "BAR"
                    // configuration of the postgres prod database.
    } } } },

From that it seems to me that there is no need for default-profile key. This would actually be a good thing since that's one key in the configuration less.

WDYT?

[hmlnarik]

I've updated the original answer.

Does that mean that the config scope of a profile encompasses both the properties defined in the profile and the ones in the root config?

Yes, exactly. In other words, the profile config inherits the root config and overrides / adds config options of that.

[sguilhen]

Ok, so you're proposing that the "empty" profile (i.e. one that gets all all the config options from the root config) is to be considered the default one?

[hmlnarik]

If by "empty" profile you mean no-profile-referenced-or-declared, then yes.

The default profile is in this proposal should be just the same as the system-wide configuration.

[sguilhen]

If by "empty" profile you mean no-profile-referenced-or-declared, then yes.

I think I need more clarification here, not sure what you meant by "no-profile-referenced-or-declared".

Currently, the algorithm to select the default profile would be as follows:

if (root config has a configured_default_profile) {
    defaultProfile = configured_default_profile;            (1)
} else {
     defaultProfile = "default";
}

Now, when we have to fetch the scope to init the factory with the right configuration options:

if (profile != "default") {
     // fetch the config from the profile with the specified name
} else if (root config has a profile named "default") {
     // fetch the config from the "default" profile
} else {
     // use the root storage factory config
}

If I understand right, you want to get rid of (1) above and not have an explicit config attribute for the default profile, but rather try to locate that based on some other rule, like an empty profile. So the algo would be something like

if (root config has an `empty` profile) {
    defaultProfile = name_of_empty_profile;
} else {
     // what goes here?? look for a profile named default as I did above? Something else?
}

I just want to understand what the algorithm to determine the profile when none is declared in one of the areas that use the map storage should look like.

In other words, if the area specifies a profile it is straightforward to locate that profile in the root config and use its config. But when the profile is not mentioned, we need to figure out what should be the default profile. I imagine you would like that to be resolved to postgres-dev in your example. We just need to define and clearly document what the rules are for configuring and resolving a default profile.

[hmlnarik]

There is no "empty" profile per se. The "default profile" is driven by the default configuration done in the same way as today for providers when we lack any profile support:

    "mapStorage": {
        ...     
        "jpa-map-storage": {
            "driver": "XYZ",
            "datasource": "FOO"
             # configuration of the postgres-dev database.
    } },

Any profile defined in a provider config would inherit all the values specified in the main provider configuration, and would be able to override those.

In other words, the profile postgres-prod in #10324 (comment) has two options:

• "datasource": "BAR" mentioned explicitly in the profile, and
• "driver": "XYZ" inherited from the main configuration of jpa-map-storage provider

[hmlnarik]

Distinguishing when to cope with configuration as reference and when as a new component

How would the code mutually distinguish the following different use cases of the configuration?

1. Reference to a predefined profile postgres-dev

    "role": {
           "provider": "map",
           "map": {
               "storage": {
                   "provider": "jpa-map-storage",
                   "provider-profile": "postgres-dev"
               }
           }
       }

2. A new configuration based on existing profile postgres-dev:

    "role": {
           "provider": "map",
           "map": {
               "storage": {
                   "provider": "jpa-map-storage",
                   "provider-profile": "postgres-dev",
                   "datasource": "FOO"
               }
           }
       }

3. A new configuration based on default profile:

    "role": {
           "provider": "map",
           "map": {
               "storage": {
                   "provider": "jpa-map-storage",
                   "datasource": "FOO"
               }
           }
       }

[sguilhen]

I'm addressing this in context of the discussion bellow, in which the config for a specific storage factory has to happen in the scope of that factory's name:

1- In this case, code builds a componentId that looks like mapStorage-f:ROOT:jpa-map-storage:postgres-dev. Any other area that references the same provider/profile will end up with the same componentId, as desired.

2- The config would actually be (according to what I've shown in the discussion bellow)

 "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "provider-profile": "postgres-dev",
                "jpa-map-storage": {
                    "datasource": "FOO"
                }
            }
        }
    }

So the code here checks if the config scope for the storage factory is not empty in the area configuration. In this case, it is not, it is defining the datasource, so we treat it as a new config that will end up having a componentId like mapstorage-f:ROOT:jpa-map-storage:RoleModel-inline, which is different from the #1 option above and would result in a completely separate factory being used to handle the roles. The scope used to init this factory is one that has the attributes defined in the role config, falling back to the postgres-dev profile.

3- Again based on the discussion bellow, the datasource config attribute would be nested in the jpa-map-storage. Again, because we have area-specific config we create a componentId described in the example above, so it will be handled by a different factory. The difference here is that we need an extra step to figure out what is the default profile so we can properly initialize the scope that will be fed to this factory in it's init method. Once we know what the default profile is, the final scope is one that has the attributes defined in the role config, falling back to the default profile's scope.

[hmlnarik]

Compatibility with component configuration

Re: Area-specific storage provider configuration

The following inline config

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "provider-profile": "postgres-prod",
                "jpa-map-storage": {
                        // alternative configuration
                }
            }
        }
    }

Would use the inline config and fallback to the postgres-prod profile.

storage declaration should be possible to store within component configuration. Component configuration cannot store trees, hence nesting in jpa-map-storage would be done by including the scope in the option key, and would be stored in a component representing the storage as:

provider=jpa-map-storage
provider-profile=postgres-prod
jpa-map-storage.option1=value1
jpa-map-storage.option2=value2
...

This means that for the respective provider factory, rather than the option1 would be invisible and jpa-map-storage.option1 is not expected in the ProviderFactory.init(Scope) method.

That is also the reason why the storage component configuration is flattened in map storage and behaves as if that was like this:

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "provider-profile": "postgres-prod",
                "option1": "value1"
                // alternative configuration
    } } } }

In this case, we check if there is an inline configuration for the storage and if we have one, we can generate a componentId just for the specific area. Something like mapStorage:jpa-map-storage:role-inline. As a result it would have its own storage factory.

How exactly would the algorithm for generation of this componentId look like? How would it get all the name components, specifically the role part?

[sguilhen]

That doesn't sound right to me. Indeed, the storage config is flattened in the map storage, but when we try to fetch the storage provider factory the code in KeycloakModelUtils.ScopeComponentModel adjusts the scope to the name of the storage provider:

https://github.com/keycloak/keycloak/blob/main/server-spi-private/src/main/java/org/keycloak/models/utils/KeycloakModelUtils.java#L328

For example, this is how the client config looked like when we were implementing this area's JPA storage (we actually used this config to test the implementation manually, we didn't use the mapStorage root configuration:

sguilhen@6b6abe3#diff-0396a8b7dd0fd795fe8e23759f0b301cd2496fcc8c558d61c5766352372f227aR72

So it does indeed adjust the scope and the property jpa-map-storage.option1 from the example above would be read by the factory as option1, as it is supposed to do.

How exactly would the algorithm for generation of this componentId look like? How would it get all the name components, specifically the role part?

I can't really get to the role part as that is an SPI name that is not available when we look for the provider. But we can use the modelType that the factories have access to. See this part in the proposed draft PR.

Thinking about it now, we could simply use the fully-qualified class name of the model as the key, so we would have a componentId that would look something like mapStorage-f:ROOT:jpa-map-provider:org.keycloak.models.RoleModel, which would be unique already without the need to concat -inline to the name.

[hmlnarik]

I can't really get to the role part as that is an SPI name that is not available when we look for the provider. But we can use the modelType that the factories have access to. See this part in the proposed draft PR.

Thinking about it now, we could simply use the fully-qualified class name of the provider as the key, so we would have a componentId that would look something like mapStorage-f:ROOT:jpa-map-provider:org.keycloak.models.RoleModel, which would be unique already without the need to concat -inline to the name.

This is right, I want to stress that this means that any provider factory that offers such an inline configuration must also provide a contextual identifier (org.keycloak.models.RoleModel in this case, though it could be shortened by e.g. org.keycloak.models.map.storage.ModelEntityUtil.getModelName() method).

[sguilhen]

I like the suggestion to use the model type mapped to model name.

[hmlnarik]

Profile referencing

One idea to consider: Would it be reasonable to reference the profile by standard path convention, i.e. provider-name/provider-profile? I.e. rather than

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage",
                "provider-profile": "postgres-prod"
    } } } }

to use rather:

  "role": {
        "provider": "map",
        "map": {
            "storage": {
                "provider": "jpa-map-storage/postgres-prod",
    } } } }

?

[sguilhen]

+1 to that. Actually, this was my first idea but I couldn't find any other code that uses this convention so I thought it would be preferrable to use a separate config attribute, but I'm totally in favor of this approach.

[sguilhen]

@stianst @pedroigor @mposolda I think it would be good if you guys could weigh in on this

[pedroigor]

@hmlnarik @sguilhen The proposal about using profiles sounds like a good idea to solve the original problem with the componentId as well as make the storage configuration simpler.

From a distribution perspective, I can think of two possible configuration approaches:

• Leverage extensions to plug the different types of providers (e.g.: jpa and ldap) and their respective options.
• Use a similar approach to how the cache is configured

I'm not 100% sure about the #1 above because the storage configuration is dynamic in the sense that users can define their own providers and name profiles accordingly, making it harder to keep it aligned with other options in terms of help message and auto-completion. Another problem with this approach is that we don't have yet discussed the outline for extensions.

The #2 approach is what I think is simpler, with a good UX, and flexible enough to support all the configuration aspects. By following this approach we could have a configuration namespace storage and use it to choose between defaults with the most common configuration settings for the new store or point to a user-defined configuration file.

For instance, if you want a single database for all areas we could still leverage the existing options to replace values in the default configuration.

It should also be possible to dynamically create data sources based on the storage configuration if users define different profiles for the JPA storage. This one, in particular, should make it easier when you need to use your own storage implementation to integrate with a legacy identity store for users, roles, and groups.

The same goes for a top-level LDAP configuration, to which we could provide specific options for configuring LDAP.

I'm not 100% sure yet about other potential use cases and combinations that make sense to provide out of the box via de the default configuration.

I'm also not 100% about how far people are going to use the top-level configuration to configure the store rather than configuring the storage on a per-realm basis.