Introduction
Keycloak is an open source identity and access system (IAM) that helps manage authentication and authorization in modern applications using standard protocols such as OpenID Connect, OAuth 2.0, and SAML 2.0. In production environments, it is essential to deploy it in high availability to prevent a single point of failure from leaving all the applications that depend on it without access.
In this manual we will configure a complete Keycloak architecture in high availability with:
- Cluster etcd 3 nodes for distributed coordination
- PostgreSQL in HA with Patroni (3 nodes) as data backend
- 2 nodes Keycloak with session replication via Infinispan
- 2 servers Nginx as load balancers with Keepalived (VIP)
The addressing table used throughout the manual is as follows:
| Role | hostname | IP |
|---|---|---|
| etcd + Patroni + PostgreSQL node1 | db01 | 192.168.1.214 |
| etcd + Patroni + PostgreSQL node2 | db02 | 192.168.1.215 |
| etcd + Patroni + PostgreSQL node3 | db03 | 192.168.1.216 |
| Keycloak node1 | kc01 | 192.168.1.217 |
| Keycloak node2 | kc02 | 192.168.1.218 |
| Nginx + Keepalived node1 | lb01 | 192.168.1.219 |
| Nginx + Keepalived node2 | lb02 | 192.168.1.220 |
| VIP (Virtual IP) | — | 192.168.1.221 |
Prerequisites
To follow this manual we need:
- 3 Debian servers for database (db01, db02, db03)
- 2 Debian servers for Keycloak (kc01, kc02)
- 2 Debian servers for Nginx/Keepalived (lb01, lb02)
- Java 17+ on Keycloak nodes
- root/sudo access on all nodes
- Network connectivity between all nodes
The ports that must be open between the different levels of the architecture:
Puerto(s) Servicio Dirección
----------- ------------------ ----------------------------------
2379/tcp etcd client API entre nodos db
2380/tcp etcd peer entre nodos db
5432/tcp PostgreSQL db nodes ↔ Keycloak nodes
8008/tcp Patroni REST API entre nodos db
8080/tcp Keycloak HTTP lb nodes → kc nodes
8443/tcp Keycloak HTTPS lb nodes → kc nodes
7800/tcp JGroups / Infinispan entre nodos kc
80/tcp Nginx HTTP clientes → lb nodes
443/tcp Nginx HTTPS clientes → lb nodes
VRRP (112) Keepalived entre lb nodesInstalling the database in HA
3.1 Package installation (on the 3 db nodes)
We execute in db01, db02 and db03:
sudo apt update && sudo apt install -y postgresql postgresql-contrib patroni etcd-server etcd-client curl3.2 etcd configuration
We configure etcd on each node by editing /etc/default/etcd. Each node has its own listening and advertising IP address.
NODE1—db01 (192.168.1.214):
# /etc/default/etcd — db01
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.214:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.214:2379"
ETCD_INITIAL_CLUSTER="node1=http://192.168.1.214:2380,node2=http://192.168.1.215:2380,node3=http://192.168.1.216:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_NAME="node1"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LOG_LEVEL="info"
ETCD_ENABLE_V2="true"NODE2—db02 (192.168.1.215):
# /etc/default/etcd — db02
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.215:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.215:2379"
ETCD_INITIAL_CLUSTER="node1=http://192.168.1.214:2380,node2=http://192.168.1.215:2380,node3=http://192.168.1.216:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_NAME="node2"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LOG_LEVEL="info"
ETCD_ENABLE_V2="true"NODE3—db03 (192.168.1.216):
# /etc/default/etcd — db03
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.216:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.216:2379"
ETCD_INITIAL_CLUSTER="node1=http://192.168.1.214:2380,node2=http://192.168.1.215:2380,node3=http://192.168.1.216:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_NAME="node3"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LOG_LEVEL="info"
ETCD_ENABLE_V2="true"We enable and start etcd on the 3 nodes:
sudo systemctl enable etcd
sudo systemctl restart etcd3.3 Verification of etcd
We check that the etcd cluster is healthy from any node:
etcdctl --endpoints=http://192.168.1.214:2379,http://192.168.1.215:2379,http://192.168.1.216:2379 endpoint healthExpected output:
http://192.168.1.214:2379 is healthy: successfully committed proposal: took = 3.2ms
http://192.168.1.215:2379 is healthy: successfully committed proposal: took = 3.5ms
http://192.168.1.216:2379 is healthy: successfully committed proposal: took = 3.8ms3.4 Patroni Configuration
We create the Patroni directory and configuration file on each node. Patroni will rely on etcd as DCS (Distributed Configuration Store).
NODE1—db01 (192.168.1.214)— /etc/patroni/config.yml:
scope: keycloak-cluster
namespace: /db/
name: node1
restapi:
listen: 192.168.1.214:8008
connect_address: 192.168.1.214:8008
etcd:
hosts:
- 192.168.1.214:2379
- 192.168.1.215:2379
- 192.168.1.216:2379
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
wal_level: replica
hot_standby: "on"
max_wal_senders: 10
max_replication_slots: 10
wal_log_hints: "on"
initdb:
- encoding: UTF8
- data-checksums
pg_hba:
- host replication replicator 192.168.1.214/32 md5
- host replication replicator 192.168.1.215/32 md5
- host replication replicator 192.168.1.216/32 md5
- host all all 192.168.1.0/24 md5
users:
admin:
password: admin_password
options:
- createrole
- createdb
postgresql:
listen: 192.168.1.214:5432
connect_address: 192.168.1.214:5432
data_dir: /var/lib/postgresql/14/main
bin_dir: /usr/lib/postgresql/14/bin
pgpass: /tmp/pgpass0
authentication:
replication:
username: replicator
password: replicator_password
superuser:
username: postgres
password: postgres_password
rewind:
username: rewind_user
password: rewind_password
tags:
nofailover: false
noloadbalance: false
clonedfrom: false
nosync: falseNODE2—db02 (192.168.1.215)— /etc/patroni/config.yml:
scope: keycloak-cluster
namespace: /db/
name: node2
restapi:
listen: 192.168.1.215:8008
connect_address: 192.168.1.215:8008
etcd:
hosts:
- 192.168.1.214:2379
- 192.168.1.215:2379
- 192.168.1.216:2379
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
wal_level: replica
hot_standby: "on"
max_wal_senders: 10
max_replication_slots: 10
wal_log_hints: "on"
initdb:
- encoding: UTF8
- data-checksums
pg_hba:
- host replication replicator 192.168.1.214/32 md5
- host replication replicator 192.168.1.215/32 md5
- host replication replicator 192.168.1.216/32 md5
- host all all 192.168.1.0/24 md5
users:
admin:
password: admin_password
options:
- createrole
- createdb
postgresql:
listen: 192.168.1.215:5432
connect_address: 192.168.1.215:5432
data_dir: /var/lib/postgresql/14/main
bin_dir: /usr/lib/postgresql/14/bin
pgpass: /tmp/pgpass0
authentication:
replication:
username: replicator
password: replicator_password
superuser:
username: postgres
password: postgres_password
rewind:
username: rewind_user
password: rewind_password
tags:
nofailover: false
noloadbalance: false
clonedfrom: false
nosync: falseNODE3—db03 (192.168.1.216)— /etc/patroni/config.yml:
scope: keycloak-cluster
namespace: /db/
name: node3
restapi:
listen: 192.168.1.216:8008
connect_address: 192.168.1.216:8008
etcd:
hosts:
- 192.168.1.214:2379
- 192.168.1.215:2379
- 192.168.1.216:2379
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
wal_level: replica
hot_standby: "on"
max_wal_senders: 10
max_replication_slots: 10
wal_log_hints: "on"
initdb:
- encoding: UTF8
- data-checksums
pg_hba:
- host replication replicator 192.168.1.214/32 md5
- host replication replicator 192.168.1.215/32 md5
- host replication replicator 192.168.1.216/32 md5
- host all all 192.168.1.0/24 md5
users:
admin:
password: admin_password
options:
- createrole
- createdb
postgresql:
listen: 192.168.1.216:5432
connect_address: 192.168.1.216:5432
data_dir: /var/lib/postgresql/14/main
bin_dir: /usr/lib/postgresql/14/bin
pgpass: /tmp/pgpass0
authentication:
replication:
username: replicator
password: replicator_password
superuser:
username: postgres
password: postgres_password
rewind:
username: rewind_user
password: rewind_password
tags:
nofailover: false
noloadbalance: false
clonedfrom: false
nosync: falseWe stop PostgreSQL (Patroni will manage it completely) and start Patroni on all 3 nodes:
sudo systemctl stop postgresql
sudo systemctl enable patroni
sudo systemctl start patroni3.5 Patroni cluster verification
patronictl -c /etc/patroni/config.yml listExpected output (node1 as leader, node2 and node3 as replicas):
+ Cluster: keycloak-cluster (7234567890123456789) ------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+--------+------------------+---------+---------+----+-----------+
| node1 | 192.168.1.214:5432 | Leader | running | 1 | |
| node2 | 192.168.1.215:5432 | Replica | running | 1 | 0 |
| node3 | 192.168.1.216:5432 | Replica | running | 1 | 0 |
+--------+------------------+---------+---------+----+-----------+3.6 Creation of the Keycloak database
We connect to the leader node (in this example node1/db01) and create the database and user for Keycloak:
sudo -u postgres psql
CREATE USER keycloak WITH PASSWORD 'keycloak_password';
CREATE DATABASE keycloak OWNER keycloak;
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak;
\qNext we add the necessary entries in pg_hba.conf so that Keycloak nodes can connect (if you haven't already added them in the section pg_hba from Patroni's bootstrap):
# Añadir al final de pg_hba.conf (gestionado por Patroni via patronictl edit-config o directamente)
host keycloak keycloak 192.168.1.217/32 md5
host keycloak keycloak 192.168.1.218/32 md5
# Recargar la configuración
patronictl -c /etc/patroni/config.yml reload keycloak-clusterKeycloak installation (on kc01 and kc02)
4.1 Java 17 installation
Keycloak requires Java 17 or higher. We install it on both Keycloak nodes:
sudo apt update && sudo apt install -y openjdk-17-jdk
java -versionExpected output:
openjdk version "17.0.11" 2024-04-16
OpenJDK Runtime Environment (build 17.0.11+9-Debian-1)
OpenJDK 64-Bit Server VM (build 17.0.11+9-Debian-1, mixed mode, sharing)4.2 Download and install Keycloak
We download the official Keycloak distribution, extract it in /opt and we create the system user:
export KC_VERSION=24.0.4
cd /opt
sudo wget https://github.com/keycloak/keycloak/releases/download/${KC_VERSION}/keycloak-${KC_VERSION}.tar.gz
sudo tar xzf keycloak-${KC_VERSION}.tar.gz
sudo ln -s /opt/keycloak-${KC_VERSION} /opt/keycloak
sudo groupadd keycloak
sudo useradd -r -g keycloak -d /opt/keycloak -s /sbin/nologin keycloak
sudo chown -R keycloak:keycloak /opt/keycloak-${KC_VERSION}4.3 Configuring Keycloak for production
We edit /opt/keycloak/conf/keycloak.conf on both nodes. The database URL points to the VIP (192.168.1.221) which will always resolve to the active PostgreSQL node via HAProxy or the Patroni balancer; alternatively we can directly use the IP of the current leader. The most robust thing is to deploy HAProxy either pgBouncer pointing to the Patroni REST API, but to simplify the manual we use the VIP:
# Base
hostname=keycloak.example.com
http-enabled=true
http-port=8080
proxy-headers=xforwarded
# Database
db=postgres
db-url=jdbc:postgresql://192.168.1.221:5432/keycloak
db-username=keycloak
db-password=keycloak_password
# Cluster / Infinispan
cache=ispn
cache-stack=tcp
# Health & Metrics
health-enabled=true
metrics-enabled=true4.4 Infinispan cluster configuration (JGroups TCP)
Keycloak uses Infinispan to replicate user sessions between nodes. We configure the TCP stack with TCPPING for static discovery. We edit /opt/keycloak/conf/cache-ispn.xml and we locate the TCP stack section to replace the discovery protocol with TCPPING:
<!-- Dentro de <stack name="tcp"> -->
<TCP bind_addr="match-interface:eth0"
bind_port="7800"
recv_buf_size="20000000"
send_buf_size="640000"
max_bundle_size="64000" />
<TCPPING initial_hosts="192.168.1.217[7800],192.168.1.218[7800]"
ergonomics="false"
port_range="0" />
<MERGE3 min_interval="10000"
max_interval="30000" />
<FD_SOCK />
<FD_ALL timeout="60000" interval="15000" />
<VERIFY_SUSPECT timeout="5000" />
<BARRIER />
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000"
max_bytes="4000000" />
<pbcast.GMS print_local_addr="true"
join_timeout="2000" />
<UFC max_credits="2000000"
min_threshold="0.4" />
<MFC max_credits="2000000"
min_threshold="0.4" />
<FRAG2 frag_size="60000" />4.5 Build and first boot
Before starting Keycloak in production mode we must execute the command build To compile the configuration in an optimized way:
sudo -u keycloak /opt/keycloak/bin/kc.sh buildFirst boot to create the administrator user (only necessary the first time on one of the nodes):
sudo -u keycloak KEYCLOAK_ADMIN=admin KEYCLOAK_ADMIN_PASSWORD=admin \
/opt/keycloak/bin/kc.sh start --optimizedOnce started correctly, we stop with Ctrl+C and we proceed to configure the systemd service.
4.6 systemd service
We create the unit file /etc/systemd/system/keycloak.service:
[Unit]
Description=Keycloak IAM
After=network.target
[Service]
Type=exec
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable keycloak
sudo systemctl start keycloak4.7 Keycloak cluster verification
We verify that Keycloak responds correctly and that the two nodes have formed the Infinispan cluster:
# Health endpoint (desde cualquier nodo o el balanceador)
curl -s http://192.168.1.217:8080/health | python3 -m json.tool
curl -s http://192.168.1.218:8080/health | python3 -m json.tool
# Verificar que el clúster Infinispan tiene 2 miembros (en los logs de Keycloak)
sudo journalctl -u keycloak -f | grep -i "cluster\|infinispan\|jgroups\|members"In the logs you should see something similar to:
INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN:
[kc01|1] (2) [kc01, kc02]Nginx Load Balancer with Keepalived
5.1 Installation (in lb01 and lb02)
sudo apt update && sudo apt install -y nginx keepalived5.2 Nginx configuration
We create the file /etc/nginx/conf.d/keycloak.conf on both balancers. We use least_conn to distribute the load evenly and we configure proxy headers so that Keycloak knows the real IP of the client:
upstream keycloak_backend {
least_conn;
server 192.168.1.217:8080 max_fails=3 fail_timeout=30s;
server 192.168.1.218:8080 max_fails=3 fail_timeout=30s;
keepalive 32;
}
server {
listen 80;
server_name keycloak.example.com;
location / {
proxy_pass http://keycloak_backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
}
location /health {
proxy_pass http://keycloak_backend/health;
proxy_set_header Host $host;
access_log off;
}
}
We check the configuration and reload Nginx:
sudo nginx -t && sudo systemctl reload nginx5.3 Keepalived configuration
Keepalived implements VRRP to move the virtual IP (VIP 192.168.1.221) automatically to the active node. First we create the Nginx health check script:
cat > /etc/keepalived/check_nginx.sh <<'EOF'
#!/bin/bash
if ! systemctl is-active --quiet nginx; then
exit 1
fi
if ! curl -sf http://127.0.0.1/health > /dev/null 2>&1; then
exit 1
fi
exit 0
EOF
chmod +x /etc/keepalived/check_nginx.shMASTER — lb01 (192.168.1.219) — /etc/keepalived/keepalived.conf:
global_defs {
router_id LB01
script_user root
enable_script_security
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 2
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass keycloak_vrrp_secret
}
virtual_ipaddress {
192.168.1.221/24
}
track_script {
chk_nginx
}
}BACKUP — lb02 (192.168.1.220) — /etc/keepalived/keepalived.conf:
global_defs {
router_id LB02
script_user root
enable_script_security
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -20
fall 2
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass keycloak_vrrp_secret
}
virtual_ipaddress {
192.168.1.221/24
}
track_script {
chk_nginx
}
}We enable and start Keepalived on both balancers:
sudo systemctl enable keepalived
sudo systemctl start keepalived5.4 Checking the balancer
We verify that the VIP is assigned to lb01 and that access to Keycloak works through it:
# Verificar que la VIP está en lb01
ip addr show eth0 | grep 192.168.1.221
# Verificar acceso a Keycloak a través de la VIP
curl -s http://192.168.1.221/health | python3 -m json.toolExpected output of the health check:
{
"status": "UP",
"checks": [
{
"name": "Keycloak readiness check",
"status": "UP"
}
]
}High availability testing
Once the installation is complete, it is essential to verify that the system correctly tolerates failures at each layer.
Test 1: failure of a Keycloak node
# En kc01: detener Keycloak
sudo systemctl stop keycloak
# Desde un cliente externo: verificar que el acceso sigue funcionando vía VIP
curl -s http://192.168.1.221/health
# Verificar que Nginx está enviando tráfico solo a kc02
sudo tail -f /var/log/nginx/access.log
# Restaurar
sudo systemctl start keycloakActive user sessions must be maintained by Infinispan replication between kc01 and kc02.
Test 2: PostgreSQL/Patroni leader failover
# Forzar un failover de Patroni (desde cualquier nodo db)
patronictl -c /etc/patroni/config.yml failover keycloak-cluster --master node1 --force
# Verificar que un nuevo líder ha sido elegido
patronictl -c /etc/patroni/config.yml list
# Verificar que Keycloak sigue respondiendo
curl -s http://192.168.1.221/healthPatroni will automatically elect a new replica as the leader in less than 30 seconds (value of ttl).
Test 3: MASTER balancer failure (lb01)
# En lb01: detener Keepalived para simular el fallo
sudo systemctl stop keepalived
# En lb02: verificar que la VIP ha migrado
ip addr show eth0 | grep 192.168.1.221
# Verificar que el acceso a Keycloak sigue funcionando
curl -s http://192.168.1.221/health
# Restaurar lb01
sudo systemctl start keepalivedKeepalived will detect the loss of lb01's VRRP announcements and lb02 will assume the role of MASTER, assigning the VIP in less than 3 seconds.
Security considerations
For a robust deployment in production, keep the following recommendations in mind:
- TLS at all layers- Enable HTTPS in Nginx (Let's Encrypt or corporate certificate), TLS encryption in etcd (
ETCD_CERT_FILE,ETCD_KEY_FILE), in PostgreSQL (ssl = on) and in Keycloak-proxy communications. - Strong passwords- Immediately change all sample passwords used in this manual (
keycloak_password,replicator_password,postgres_password,keycloak_vrrp_secret). Use a secrets manager like HashiCorp Vault or AWS Secrets Manager. - Firewall rules- Implements strict rules between layers. Keycloak nodes should only accept connections from balancers on port 8080. Database nodes should only accept connections from Keycloak nodes on port 5432. etcd ports (2379/2380) should only be accessible between db nodes.
- Administration console: Restrict access to
/auth/admin/by IP using a policyallow/denyin Nginx, or use a VPN. - PostgreSQL backups- Set up regular backups with
pg_basebackupeitherpgBackRest. Patroni is not a substitute for backups: it protects against failures, not human errors. - Monitoring- exposes Keycloak metrics (
metrics-enabled=true) to Prometheus and visualize them in Grafana. Monitor the status of the Patroni cluster withpatroni_exporterand etcd state with its own exporter. - Log rotation: configure
logrotatefor Nginx and Keycloak logs to avoid disk space exhaustion.
:wq!
Comments