Deprecated: HotRod Map Storage: How To and Q&A

[mhajas]

Setup Keycloak server with the Infinispan server as storage

Setting up Infinispan server

Setup of an Infinispan server using either docker or podman:

docker run -it -p 11222:11222 -e USER="admin" -e PASS="admin" quay.io/infinispan/server:14.0.9.Final

Managing Infinispan server

There are two ways how to manage the Infinispan server. The first one is Infinispan Server management console; sign in with username and password admin.

The other option is Infinispan CLI, available here: https://github.com/infinispan/infinispan-quarkus/releases/tag/13.0.10.Final

In the zip file there is infinispan-cli executable file. After executing a CLI environment is opened.

For connecting to our Infinispan server one can write connect command which asks for a username and password (both admin).

Here is a link to CLI docs for more details.

Removing caches with CLI

• Keycloak creates one cache for each area. It is possible to display all caches by the following commands:

cd caches
ls

• To remove all entries from a cache my-cache do the following:

clearcache my-cache

• or to remove the cache completely do:

drop my-cache

NOTE: it is also possible to remove caches in the Server management console.

Running Keycloak with HotRod as its storage

WARNING: with the setup below, the Keycloak test server will only use the HotRod Storage for entities that have the storage implemented, and for others will use the concurrent hash map storage that stores information in target/map (No entities like this at the moment, but there will be events and singleUseObjects shortly).

• Clearing data from the concurrent hash map storage (needs to be cleared if Infinispan server is being cleared)

rm -rf target/map

• Build Keycloak (needed when there have been changes to Keycloak source code)

mvn clean install -DskipTests

• Run org.keycloak.testsuite.KeycloakServer with the following parameters; the given properties will replace values in https://github.com/keycloak/keycloak/blob/main/testsuite/utils/src/main/resources/META-INF/keycloak-server.json

mvn -pl testsuite/utils \
-Pkeycloak-server,map-storage \
-Dkeycloak.authSession.map.storage.provider=hotrod \
-Dkeycloak.client.map.storage.provider=hotrod \
-Dkeycloak.clientScope.map.storage.provider=hotrod \
-Dkeycloak.role.map.storage.provider=hotrod \
-Dkeycloak.realm.map.storage.provider=hotrod \
-Dkeycloak.group.map.storage.provider=hotrod \
-Dkeycloak.user.map.storage.provider=hotrod \
-Dkeycloak.authorization.map.storage.provider=hotrod \
-Dkeycloak.userSession.map.storage.provider=hotrod \
-Dkeycloak.loginFailure.map.storage.provider=hotrod \
-Dkeycloak.singleUseObject.map.storage.provider=hotrod \
-Dkeycloak.adminEventsStore.map.storage.provider=hotrod \
-Dkeycloak.authEventsStore.map.storage.provider=hotrod \
-Dkeycloak.connectionsHotRod.embedded=false \
-Dkeycloak.connectionsHotRod.configureRemoteCaches=true \
-Dkeycloak.connectionsHotRod.port=11222 \
-Dkeycloak.connectionsHotRod.enableSecurity=true \
-Dkeycloak.connectionsHotRod.reindexCaches=all \
exec:java

• NOTE: Keycloak is reindexing caches when it is starting. This is done because the indexes live outside of the Infinispan server (in filesystem) and are not removed when caches are removed from the server. If we start in a clean environment, we can omit keycloak.connectionsHotRod.reindexCaches parameter which will speed up the startup.
• After the startup command, Keycloak should create all caches necessary for running with the Infinispan server.
• You can confirm that everything was created either in Server management console, or in infinispan cli, for example:

cd caches/realms
ls

• This ^^ should output a UUID of the master realm that was created during startup. To see master realm details, execute the following (replace [UUID] with the output of the previous command):

get [UUID]

Alternative setup with Quarkus

After building Keycloak, run

mvn -f quarkus/server/pom.xml compile quarkus:dev -Dquarkus.args="start-dev --storage=hotrod --storage-hotrod-host=localhost --storage-hotrod-port=11222 --storage-hotrod-username=admin --storage-hotrod-password=admin" -Dkc.config.args="start-dev;;--storage=hotrod;;"

Or use the Keycloak nightly distribution or unpack the locally build file quarkus/dist/target/keycloak-999-SNAPSHOT.zip and run

bin/kc.sh start-dev --storage=hotrod --storage-hotrod-host=localhost --storage-hotrod-port=11222 --storage-hotrod-username=admin --storage-hotrod-password=admin

Index data storing

By default, Infinispan stores index data in persistent storage instead of memory. The data stored in the caches are stored in memory. This can lead to inconsistency, for example, after container restart when indexed data survive restarting while cached data does not.

To avoid this Keycloak configures indexed data to be stored in memory too using the following cache config.

<distributed-cache name="users" mode="SYNC">
    ....
    <indexing storage="local-heap">
        <indexed-entities>
            ....
        </indexed-entities>
    </indexing>
</distributed-cache>

The relevant part is storage="local-heap".

Known issues and limitations

Summary:

• Keycloak currently doesn't support newer versions of Infinispan ( > 12 ). It fails on startup when Keycloak tries to create caches. As a workaround, it should be possible to create all necessary caches manually and then start Keycloak. Beware this workaround was not thoroughly tested. - Update Infinispan version to 14.x.x #12494
• No-downtime upgrade requires Infinispan 14 - Update Infinispan version to 14.x.x #12494
• On startup Keycloak sometimes reindex caches, due to this it can take more time to start Keycloak. In future, we plan to avoid reindexing as it breaks no-downtime upgrade - Stop reindexing indexes on new version #12497
• Currently, we are not able to upgrade Infinispan cache configurations in a no-downtime way - Investigate cache config update on existing cache #12265
• Infinispan doesn't support bulk removal in version 12, therefore removal of a high amount of entities may be slow - Use bulk deletes in HotRod store #9420

Open issues

[ahus1]

Hi @mhajas - I ran the Infinispan container via podman, and Keycloak Quarkus with Maven, and then got the following Exception continuously:

2022-07-21 11:15:25,706 WARN  [org.infinispan.HOTROD] (HotRod-client-async-pool-1-8) ISPN004105: Reverting to the initial server list for caches [___protobuf_metadata]
2022-07-21 11:15:25,859 INFO  [org.infinispan.HOTROD] (HotRod-client-async-pool-1-10) ISPN004006: Server sent new topology view (id=1, age=0) containing 1 addresses: [10.0.2.100:11222]
2022-07-21 11:15:25,859 INFO  [org.infinispan.HOTROD] (HotRod-client-async-pool-1-10) ISPN004014: New server added(10.0.2.100:11222), adding to the pool.
2022-07-21 11:15:25,860 INFO  [org.infinispan.HOTROD] (HotRod-client-async-pool-1-10) ISPN004016: Server not in cluster anymore(localhost:11222), removing from the pool.
2022-07-21 11:15:25,996 WARN  [org.infinispan.HOTROD] (HotRod-client-async-pool-1-11) ISPN004105: Reverting to the initial server list for caches [___protobuf_metadata]
2022-07-21 11:15:25,995 WARN  [org.infinispan.HOTROD] (HotRod-client-async-pool-1-23) ISPN004015: Failed adding new server 10.0.2.100:11222: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /10.0.2.100:11222

It seems to me that Infinispan advertises 10.0.2.100 as its public IP address, which is then unavailable for Keycloak to reach. Can I force Infinispan to advertise itself from a specific IP address (localhost)?

[mhajas]

Hello, I have seen this already. The problem is exactly what you described Infinispan advertises itself on a different IP address than localhost. I don't know how to configure ISPN to advertise different IP. I would say it is possible using some custom config (it seems something like this could help)

The simplest solution to avoid this error is to configure Keycloak with --storage-hotrod-host=IP_ADDRESS instead of localhost (I needed to run docker image with --network host to make the IP visible to Keycloak). I used localhost because it is simpler and it works.

[ahus1]

OK, the --network host solved it for me.

So I now ran the pod this way (as a daemon, and with a name, but that's personal preference ... the port mapping is removed as this doesn't seem to be available for for the host network):

podman run -d --network host --name infinispan -e USER="admin" -e PASS="admin" quay.io/infinispan/server:12.1.7.Final

Then looked up the IP address the cache used for binding:

netstat -nl | grep 11222

Finally use the IP it showed as a parameter for storage-hotrod-host.

Then it worked for me. Maybe you could pick and choose some of the parts for the description above.

[alexted]

Is this information available somewhere in the official documentation?

[mhajas]

@alexted not yet. This stuff is currently released only as a preview so we didn't include it in the official docs yet. Why are you asking? Are you missing some info we should add here?

[dominikschlosser]

Is there a way to completely disable cache-reindexing? Right now there seems to be a value 'all', which is the default, but no value for 'none'. So omitting the property doesnt help because then 'all' is used.

[mhajas]

The reason for setting it to all by default is that at the moment no-downtime upgrade doesn't work without it. We will get rid of this entirely when we upgrade to Infinispan 14.

If you want to disable it, it should be possible to use this: -Dstorage-hotrod-cache-reindex="" (theoretically, you should be able to use any string that doesn't match any existing cache name and Keycloak should not reindex caches).

Out of curiosity, why do you want to disable it? Is it slow for you? How many entries do you have in caches?

[kkcmadhu]

out of curiocity, is the performance of this new storage with hotrod compare with the legacy store?
from what i see that the hotrod based store is having severe latency check these out
#15174
#14984

[mhajas]

@kkcmadhu Thank you very much for trying out the new store. Please keep in mind that it is in preview mode only. The performance cannot be compared with the legacy store yet.

The new store still requires some work, especially around caching (currently, there is no caching layer enabled on the Keycloak server so all data are loaded from the store each time they are requested). I am sure more details about caching in the new store will be shared in the future.

On the other hand, the performance drop you described is quite significant, we will have a look at it.

btw. if you are interested in how we are doing performance testing, you can have a look at keycloak-benchmark repository.

[kkcmadhu]

also, presently i think there is no way to reuse an infinispan cluster for multiple keycloak deployments.
i think we will need to support dynamic cache names, adding one infinispan cluster , per keycloak deployment may not the best option..

#15007

[ahus1]

As announced to the community in this blog post, we are no longer developing the Map Store, and it will be removed from Keycloak's codebase. Follow #24111 for the removal steps.