Trouble getting OpenID Connect set up using Keycloak -- receiving error message

I am having quite a bit of trouble getting OpenID Connect set up. It all stems from a single error that I keep getting, and I can't seem to resolve the error:

[2024-03-21T13:09:57,036][ERROR][org.elasticsearch.bootstrap.Elasticsearch] [es01] fatal exception while booting Elasticsearch

java.lang.IllegalStateException: security initialization failed

at org.elasticsearch.xpack.security.Security.createComponents(Security.java:637) ~[?:?]
...
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-8.12.2.jar:?]
Caused by: org.elasticsearch.common.settings.SettingsException: The configuration setting [xpack.security.authc.realms.oidc.myRealmName.rp.client_secret] is required
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectRealm.buildRelyingPartyConfiguration(OpenIdConnectRealm.java:256) ~[?:?]

From my elasticsearch.yml:

xpack.security.authc.token.enabled: true
xpack.security.authc.realms.oidc.myRealmName:
  order: 2
  rp.client_id: "kb01"
  rp.response_type: code
  rp.redirect_uri: "https://kb01.myRealmName.com:443/api/security/oidc/callback"
  op.issuer: "https://sso.myRealmName.com/realms/myRealmName"
  op.authorization_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/auth"
  op.token_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/token"
  op.jwkset_path: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/certs"
  op.userinfo_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/userinfo"
  op.endsession_endpoint: "https://sso.myRealmName.com/realms/myRealmName/protocol/openid-connect/logout"
  rp.post_logout_redirect_uri: "https://kb01.myRealmName.com:443/logged_out"
  claims.principal: name
  ssl.verification_mode: none

from my main docker-compose.yml:

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    dns:
      - 172.24.1.2
    container_name: es01
    networks:
      default: 
      web:
        ipv4_address: 172.24.0.11
    restart: unless-stopped
    volumes:
      - ./config:/usr/share/elasticsearch/config
      - ./logs:/usr/share/elasticsearch/logs
      - ./esdata01:/usr/share/elasticsearch/data
    environment:
      - "ES_JAVA_OPTS=-Delasticsearch.xpack.security.authc.debug=true"
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key  
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt  
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.es01.entrypoints=websecure"
      - "traefik.http.routers.es01.rule=Host(`es01.myDomain.com`)"
      - "traefik.http.routers.es01.service=es01"
      - "traefik.http.services.es01.loadbalancer.server.port=9200"
      - "traefik.http.services.es01.loadbalancer.server.scheme=https"
      # - "traefik.http.services.es01.loadbalancer.serversTransport=es01"
      # - "traefik.http.services.es01.loadbalancer.passhostheader=true"
      - "traefik.http.routers.es01.tls=true"
      - "traefik.http.routers.es01.tls.certresolver=myresolver"
      # - "traefik.http.serversTransports.es01.serverName=es01"
      # - "traefik.http.serversTransports.es01.insecureSkipVerify=true"
      # - "traefik.docker.network=web"
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

from my kibana.yml:

server.name: kibana
server.host: "0.0.0.0"
elasticsearch.hosts: "https://es01.myDomain.com"
elasticsearch.username: "kibana_system"
elasticsearch.password: "randomPassword"
elasticsearch.ssl.verificationMode: "none"
xpack.security.secureCookies: true
xpack.security.sameSiteCookies: "None"
logging.root.level: info
xpack.security.authc.providers:
  oidc.myRealmName:
    order: 0
    realm: myRealmName
    description: "Keycloak"
  basic.basic:
    order: 1

At a certain point in the setup process, using a modified version of the setup script from the official documentation, I have the following:

        echo "${CLIENT_SECRET}" | bin/elasticsearch-keystore add 'xpack.security.authc.realms.oidc.myRealmName.rp.client_secret' --stdin;
        chown -R 1000:0 config/elasticsearch.keystore;
        echo "Waiting for Elasticsearch availability";

I have verified that the rp.client_secret is getting applied to the elasticsearch.keystore this way by exec'ing into the container and running the following command and getting the following results:

elasticsearch@fb2b72cabc00:~/bin$ ./elasticsearch-keystore list
keystore.seed
xpack.security.authc.realms.oidc.myRealmName.rp.client_secret

I have also changed the healthcheck in the setup container so that it doesn't report healthy until the elasticsearch-keystore commands have completed:

    healthcheck:
      test: ["CMD-SHELL", "[ -f config/elasticsearch.keystore ]"]
      interval: 1s
      timeout: 5s
      retries: 120

The revised healthcheck is working. the elasticsearch container is not deploying until the above is complete. Nonetheless, I continue to get the above error message that I have provided to you up top. I don't know what else to try.

Additional information that may be helpful:

  1. I am using Keycloak as the OIDC provider
  2. I have successfully used Keycloak with other services that I have deployed using similar setups.
  3. I am able to get the elasticsearch / kibana pair running just fine WITHOUT OIDC. It's only when I am attempting the OIDC that I am having trouble.

I appreciate any assistance you can provide.

Any takers on answering this critical question?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.