I am having quite a bit of trouble getting OpenID Connect set up. It all stems from a single error that I keep getting, and I can't seem to resolve the error:
[2024-03-21T13:09:57,036][ERROR][org.elasticsearch.bootstrap.Elasticsearch] [es01] fatal exception while booting Elasticsearch
java.lang.IllegalStateException: security initialization failed
at org.elasticsearch.xpack.security.Security.createComponents(Security.java:637) ~[?:?]
...
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-8.12.2.jar:?]
Caused by: org.elasticsearch.common.settings.SettingsException: The configuration setting [xpack.security.authc.realms.oidc.myRealmName.rp.client_secret] is required
at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectRealm.buildRelyingPartyConfiguration(OpenIdConnectRealm.java:256) ~[?:?]
From my elasticsearch.yml:
xpack.security.authc.token.enabled: true
xpack.security.authc.realms.oidc.myRealmName:
order: 2
rp.client_id: "kb01"
rp.response_type: code
rp.redirect_uri: "https://kb01.myRealmName.com:443/api/security/oidc/callback"
op.issuer: "https://sso.myRealmName.com/realms/myRealmName"
op.authorization_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/auth"
op.token_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/token"
op.jwkset_path: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/certs"
op.userinfo_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/userinfo"
op.endsession_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/logout"
rp.post_logout_redirect_uri: "https://kb01.myRealmName.com:443/logged_out"
claims.principal: name
ssl.verification_mode: none
from my main docker-compose.yml:
es01:
depends_on:
setup:
condition: service_healthy
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
dns:
- 172.24.1.2
container_name: es01
networks:
default:
web:
ipv4_address: 172.24.0.11
restart: unless-stopped
volumes:
- ./config:/usr/share/elasticsearch/config
- ./logs:/usr/share/elasticsearch/logs
- ./esdata01:/usr/share/elasticsearch/data
environment:
- "ES_JAVA_OPTS=-Delasticsearch.xpack.security.authc.debug=true"
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
labels:
- "traefik.enable=true"
- "traefik.http.routers.es01.entrypoints=websecure"
- "traefik.http.routers.es01.rule=Host(`es01.myDomain.com`)"
- "traefik.http.routers.es01.service=es01"
- "traefik.http.services.es01.loadbalancer.server.port=9200"
- "traefik.http.services.es01.loadbalancer.server.scheme=https"
# - "traefik.http.services.es01.loadbalancer.serversTransport=es01"
# - "traefik.http.services.es01.loadbalancer.passhostheader=true"
- "traefik.http.routers.es01.tls=true"
- "traefik.http.routers.es01.tls.certresolver=myresolver"
# - "traefik.http.serversTransports.es01.serverName=es01"
# - "traefik.http.serversTransports.es01.insecureSkipVerify=true"
# - "traefik.docker.network=web"
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
from my kibana.yml:
server.name: kibana
server.host: "0.0.0.0"
elasticsearch.hosts: "https://es01.myDomain.com"
elasticsearch.username: "kibana_system"
elasticsearch.password: "randomPassword"
elasticsearch.ssl.verificationMode: "none"
xpack.security.secureCookies: true
xpack.security.sameSiteCookies: "None"
logging.root.level: info
xpack.security.authc.providers:
oidc.myRealmName:
order: 0
realm: myRealmName
description: "Keycloak"
basic.basic:
order: 1
At a certain point in the setup process, using a modified version of the setup script from the official documentation, I have the following:
echo "${CLIENT_SECRET}" | bin/elasticsearch-keystore add 'xpack.security.authc.realms.oidc.myRealmName.rp.client_secret' --stdin;
chown -R 1000:0 config/elasticsearch.keystore;
echo "Waiting for Elasticsearch availability";
I have verified that the rp.client_secret is getting applied to the elasticsearch.keystore this way by exec'ing into the container and running the following command and getting the following results:
elasticsearch@fb2b72cabc00:~/bin$ ./elasticsearch-keystore list
keystore.seed
xpack.security.authc.realms.oidc.myRealmName.rp.client_secret
I have also changed the healthcheck in the setup container so that it doesn't report healthy until the elasticsearch-keystore commands have completed:
healthcheck:
test: ["CMD-SHELL", "[ -f config/elasticsearch.keystore ]"]
interval: 1s
timeout: 5s
retries: 120
The revised healthcheck is working. the elasticsearch container is not deploying until the above is complete. Nonetheless, I continue to get the above error message that I have provided to you up top. I don't know what else to try.
Additional information that may be helpful:
- I am using Keycloak as the OIDC provider
- I have successfully used Keycloak with other services that I have deployed using similar setups.
- I am able to get the elasticsearch / kibana pair running just fine WITHOUT OIDC. It's only when I am attempting the OIDC that I am having trouble.
I appreciate any assistance you can provide.