Active-passive multi-site deployments with Keycloak 23

[ahus1]

Deploying Keycloak to multiple independent sites is essential for some environments to provide high availability and a speedy recovery from failures. This release adds preview-support for active-passive deployments for Keycloak.

A lot of work has gone into testing and verifying a setup which can sustain load and recover from the failure scenarios. To get started, go to the "High Availability" section at the Keycloak Guides, or go directly to Multi-site deployments. This replaces the "Cross-DC" documentation which was available in previous releases of Keycloak.

While we're continuing to work on this, we'd be happy to get your feedback as comments to this post.

Thanks!

[ShaswataJash]

Hello @ahus1 , can you please link to the actual benchmark test page from which following results were captured under https://www.keycloak.org/high-availability/concepts-memory-and-cpu-sizing

• For each 40 user logins per second, 1 vCPU per Pod in a three-node cluster
• For each 450 client credential grants per second, 1 vCPU per Pod in a three-node cluster
• For each 350 refresh token requests per second, 1 vCPU per Pod in a three-node cluster

Under the benchmark test page, I am expecting a webpage which captures

• the benchmark commands (e.g. the exact commands from https://www.keycloak.org/keycloak-benchmark/) that were used
• the graph showing the load testing ramp up vs result
• Any other details which will help the audience to replicate the exactly same test setup from which the above mentioned dimensioning guide was written under 'concepts-memory-cpu-sizing' e.g.

1. The above dimensioning guide should be at-least normalized against CPU speed of 'm5.4xlarge' (which I think is 3.1 Ghz). The exact capture of /proc/cpuinfo from the pod where keycloak server was running, would be necessary for extrapolation.
2. Which docker image was used for keycloak server (specifically, what was the JVM version in that image, whether there was any optimization available for java-cryptographic-extension)
3. What were the features enabled for keycloak server?
4. Is there any additional environment variables enabled at Quarkus level etc.?

[ahus1]

You can find more information on this page: https://www.keycloak.org/keycloak-benchmark/benchmark-guide/latest/report/rosa-benchmark-key-results

The Keycloak Benchmark is an incubating space and more information will move to Keycloak's main repository as it matures.

About item (1) - in order to do normalization, one would need to do multiple tests on different architectures. As this is not on the immediate road map, this is unlikely to happen soon, as other activities have a higher priority.

About items (2) to (4): Assume the defaults, and be aware that Keycloak provides its own official images, so why should the benchmark use something else?

[ShaswataJash]

@ahus1 I think for the current statements in the benchmark result

• For each 40 user logins per second, 1 vCPU per Pod in a three-node cluster
• For each 450 client credential grants per second, 1 vCPU per Pod in a three-node cluster
• For each 350 refresh token requests per second, 1 vCPU per Pod in a three-node cluster

can be interpreted as -

• Thus total (40 * 3) = 120 user logins per second cumulatively for the keycloak-cluster with total 3 vCPU worth of cpu-requests and limits".
• Thus total (450 * 3) = 1350 client credential grants per second cumulatively for the keycloak-cluster with total 3 vCPU worth of cpu-requests and limits".
• Thus total (350 * 3) = 1050 refresh token requests per second cumulatively for the keycloak-cluster with total 3 vCPU worth of cpu-requests and limits".

Rather, what you may have meant -

• For each 40 user logins per second, per keycloak-cluster (deployment/statefulset) where cluster consists of three Pods with 1 vCPU per Pod
• For each 450 client credential grants per second, per keycloak-cluster (deployment/statefulset) where cluster consists of three Pods with 1 vCPU per Pod
• For each 350 refresh token requests per second, per keycloak-cluster (deployment/statefulset) where cluster consists of three Pods with 1 vCPU per Pod

Or,

• For each (40/3) user logins per second per 1 vCPU
• For each (450/3) client credential grants per second per 1 vCPU
• For each (350/3) refresh token requests per secondper 1 vCPU

Please clarify further. If my understanding is correct, then you should please reword the benchmark summary result such a way it does not keep scope of misinterpretation.

[ahus1]

The correct answer is "For each 40 user logins per second, per keycloak-cluster (deployment/statefulset) where cluster consists of three Pods with 1 vCPU per Pod", and I would suppose the example details this.

As this is an open source project, please consider creating a pull request to update the docs.

[danielgit0]

Hi @ahus1,

I am currently trying to do a Keycloak deployment with an external Infinispan. But I am getting the following error:

2023-12-19 16:56:01 2023-12-19 15:56:01,152 ERROR [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000660: DefaultCacheManager start failed, stopping any running components: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper{name=org.infinispan.globalstate.GlobalConfigurationManager, instance=org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl@33407da8, status=FAILED} failed to start

My environment details:

• local kubernetes cluster v1.28.2 provided by Docker Desktop v4.26.0.
• MySQL database, it is running in a docker container using the image: arm64v8/mysql:8.0.34.
• Infinispan v14.0.19.Final, deployed using the operator v2.3.6 following the guide from the benchmark site for a Single Cluster. It is on a separate namespace called infinispan.
• Keycloak 23.0.3 is being deployed using vanilla kubernetes. It is in a namespace called keycloak.

As there are many files and the post could get quite long, I am attaching the complete files and logs in a zip, with the following contents:

• identities.yml: infinispan credentials
• infinispan-cr.yml: k8s CR
• caches.yml: k8s CR caches definition
• custom-cache-ispn.xml: Keycloak's Infinispan configuration.
• deployment-with-infinispan.yml: k8s Keycloak vanilla deployment.

keycloak-with-external-infinispan.zip

Do you notice something wrong in my configuration?
Thanks in advance.

By the way, I did try to install Keycloak with the operator v.23.0.3 following this example, but I got the error error: no kind "Keycloak" is registered for version "k8s.keycloak.org/v2alpha1" in scheme "pkg/scheme/scheme.go:28"

[ahus1]

Thank you for the update. As you found out, the Keycloak Operator needs to run in the same Namespace where you run Keycloak. This is sadly a line at the end of the document: "Currently the Operator watches only the namespace where the Operator is installed."

The logs look good. There are actually two Infinispans clusters setup: One inside of Keycloak, and another external Infinispan.

We haven't documented to put the JARs into Infinispan, and haven't documented how to use Infinispan's console. It is not essential for running this setup to do this, we might add it in the future. As those JARs would need to match the version of Keycloak, you would need to keep them in sync, which can be a bit of a headache (or even nightmare), so we might not support it.

keycloak/keycloak-benchmark#660

[danielgit0]

You're right, maintaining those jars in sync will be a headache.

About the Keycloak operator namespace, I would have never thought about running Keycloak in the same namespace if I hadn't read it from the link of the documentation that you posted. As stated from the beginning, it is a feature under preview, so I guess that at some point that will be changed/improved.

I also got it to work with vanilla Kubernetes (NOT RECOMMENDED as the Operator takes care of this in the correct way): deployment-with-external-infinispan.yml.txt

And the sessions are now kept during restarts.

Thanks again!

[ahus1]

I see you're setting jgroups.dns.query to the service of the external Infinispan. This should point to a headless service of Keycloak so Keycloak can form a separate Infinispan cluster with its embedded Infinispan.

I also see that you start Keycloak with "start-dev", which you shouldn't do in production, but only for development.

I see a lot of problems with this approach as you're re-inventing the wheel when not using the Operator.

[danielgit0]

Thanks, I am aware that it is not ideal and the start-devwas on purpose. I just wanted to have "a quick working demo" with my initial approach for completeness.
The Keycloak Operator is the way to go, the reply was updated to prevent some misleading there.

[ahus1]

Thank you for updating the previous post!

[tsaarni]

Thanks for the guides @ahus1!

Would it be possible / worthwhile to clarify the upgrade procedure in this deployment model?

In Known limitations it is mentioned:

On Keycloak or Infinispan version upgrades (major, minor and patch), all session data (except offline session) will be lost as neither supports zero downtime upgrades.

I assume both active and passive instance must be running same versions of software (e.g. due to DB schema differences) and the upgrade procedure for active-passive deployment would follow Upgrading the Keycloak server but now applying to deployment as a whole? The deployment model would not improve upgrades to achieve (near) zero-downtime?

[ahus1]

The deployment model would not improve upgrades to achieve (near) zero-downtime?

No it doesn't. There are some cases where it can improve when you only upgrade Keycloak, and the online user sessions are still available in the external Infinispan.

In the past, patch upgrades of Keycloak usually didn't require downtimes, but that's not guaranteed.

[tsaarni]

If you think it could be useful, I'd be glad to submit a PR with clarification, a new bullet under Known limitations / Upgrades with something like:

• The upgrade process for multi-site deployments mirrors that of non-highly available deployments. This necessitates simultaneous upgrades across all sites, which implies service downtime during the upgrade period.

It could also mention about patch releases, though I wonder if there is a way for user to find out which patch releases are safe to upgrade without downtime. If there isn't, maybe it is not good to add that.

[ahus1]

From a user's point of view, this is something that should be stated in the release notes. At the moment, we don't state it there. I would need to be an extra (manual) step in our release process to determine that. I'll make a note about this, but can give a date when we'll come around for that.

[tsaarni]

No worries. I was rather just thinking that IF patch releases would be supported with no downtime, then there should be some way to find that info.

What I was actually looking for, was just to document that even in multi-site deployment we need to take both sites down and upgrade them at once. It is not possible to avoid downtime by keeping running e.g. Keycloak 22 in one site while upgrading the other one to Keycloak 23. Someone could now think this is possible.

Therefore I was suggesting another bullet point in Known limitations / Upgrades.

[ahus1]

With Keycloak 24, the A/P deployment is now fully supported and no longer preview. Enjoy!

[jwklijnsma]

@ahus1 is there any documents for more realms in your configs ?
Event object wasn't available in remote cache after event was received. Event key:

[mhajas]

@jwklijnsma Were you able to create an issue for the behavior you are experiencing? Or have you been able to solve it?

[jwklijnsma]

we solvit but the probleem is that we need restart the pods then the cache is correct so maybe keycloak needs update for that

[mhajas]

Thank you for the response @jwklijnsma. Restarting pods should not be necessary and it does not not sound like a solution. Could you please create a new bug report for this with steps to reproduce so we can check on this?

[jwklijnsma]

#27975 bug if you needs te config

[sventorben]

Hey @ahus1, kudos on the fantastic work you've done! It's definitely a significant leap forward.

I'm curious, are there any upcoming plans to implement active/active setups?

Also, I'm currently delving into designing a multi-site deployment strategy with more than two sites, incorporating both on-prem and hyperscaler environments (a hybrid cloud setup). My aim is to establish a seamless cluster across all availability zones, effectively presenting it as a singular KC system. However, it's crucial that each instance remains independent and operational if the connection between on-prem and hyperscalers experiences any disruptions.

Do you have any pointers, documentation, or insights on achieving this? Your guidance would be immensely appreciated. Thanks a lot!

[ahus1]

For an A/A setup you would require all session information to be present on the other site before the primary site returns the response to the caller. At the moment some of of the processes are still asynchronous, especially around the processing of invalidations via the work cache. So I'd say A/A is not supported at the moment, and I wouldn't recommend it. Only A/P is supported and tested for now.

For A/A we would need to complete at least #28265, #28745 and some method to prevent a split-brain between A/A as described in #29303

Regarding preview features: We're working on reducing the number of preview features one by one, and we'll eventually share a road map. As the list is quite long, it might take a while until we get to token exchange.

[ahus1]

A different setup we might eventually have earlier is an Active-Standby setup, where the standby is in a different region and gets its data via an asynchronous database replication. In this setup, you will lose some seconds of data on a failover, and need to start you nodes on the standby site once the database has the standby site has taken over. Such a setup should be possible with an Aurora Global Database. As of today such a setup would loose all online sessions, with #28265 it might even keep the sessions (although they will be some seconds out-of-date on failover due to data lost in the replication delay).

[sventorben]

For an A/A setup you would require all session information to be present on the other site before the primary site returns the response to the caller.

While theoretically true, couldn't this be addressed with some form of session stickiness in most cases?

A different setup we might eventually have earlier is an Active-Standby setup

Could you elaborate on its benefits compared to an active/passive configuration, please? I might be overlooking something here. Or are you suggesting implicitly that an active/passive setup should also not span regions?

We're working on reducing the number of preview features one by one, and we'll eventually share a road map. As the list is quite long, it might take a while until we get to token exchange.

Can I reasonably assume that it won't be deprecated but rather transitioned to a stable state eventually?

[ahus1]

While theoretically true, couldn't this be addressed with some form of session stickiness in most cases?

As long as users access Keycloak via their browser, this might be possible. Still the OIDC protocol when confidential clients are involved, they directly contact Keycloak for example to exchange the code for the token. In this scenario sticky sessions don't work, so it is an overall "no" answer to that question.

are you suggesting implicitly that an active/passive setup should also not span regions?

Yes, and we documented it here. In cross-region setups, the latency between Infinispan will grow. As for each request by Keycloak there are multiple interactions with Infinispan on the other site, the latencies will multiply. For similar reasons there are also no synchronous database setups for cross-region when you look in the AWS portfolio. Aurora with synchronous replication is only available within a single region.

Can I reasonably assume that it won't be deprecated but rather transitioned to a stable state eventually?

I know there are several people out there using it, and they would be very sad if we would do that. Still with the current state of the feature it allows us to withdraw or deprecate the feature. I won't make promises either way.

With the capacity of people helping maintaining Keycloak, we will always need to find the balance between new development and the right amount of supported (and maintained) features. Supporting more features would eventually require more people maintaining Keycloak.

[sventorben]

to exchange the code for the token

Good catch. I almost ever forget that part ;)

Thanks @ahus1 - that was really helpful.