Elasticsearch Unable to Fetch Client Secret from Keystore for OIDC Configuration (Version 7.17/8.x)

Elastic Stack Kibana
1

Hello,

I am attempting to enable Single Sign-On (SSO) in Elasticsearch using Keycloak as the identity provider, by configuring OIDC (OpenID Connect) authentication. Following the official Elasticsearch documentation for version 7.17, 8.11, and 8.13, I have added the necessary OIDC fields in the elasticsearch.yml configuration file. Despite adding the client secret to the Elasticsearch keystore via the elasticsearch-keystore CLI, Elasticsearch is still unable to retrieve the required keys from the keystore, and I am encountering the following error:

SettingsException[The configuration setting [xpack.security.authc.realms.oidc.customerdashboard.rp.client_secret] is required]

Configuration Overview:

Below is the current content of my elasticsearch.yml file:

cluster.name: "{cluster_name}"
node.name: "{node_name}"
path.data: {path_data}
path.repo: {path_repo}
path.logs: {path_logs}
network.host: [{network_host1}, {network_host2}]
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: {ssl_key_path}
xpack.security.http.ssl.certificate: {ssl_certificate_path}
xpack.security.http.ssl.certificate_authorities: {ssl_ca_path}
xpack.security.transport.ssl.key: {ssl_key_path}
xpack.security.transport.ssl.certificate: {ssl_certificate_path}
xpack.security.transport.ssl.certificate_authorities: {ssl_ca_path}
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.verification_mode: certificate
cluster.routing.allocation.enable: all
http.type: security4
xpack.security.authc.token.enabled: true
xpack.security.authc.realms.oidc.customerdashboard:
  order: 2
  rp.client_id: "{client_id}"
  rp.response_type: code
  rp.redirect_uri: "{redirect_uri}"
  op.issuer: "{issuer_url}"
  op.authorization_endpoint: "{authorization_endpoint}"
  op.token_endpoint: "{token_endpoint}"
  op.jwkset_path: "{jwkset_path}"
  op.userinfo_endpoint: "{userinfo_endpoint}"
  op.endsession_endpoint: "{endsession_endpoint}"
  rp.post_logout_redirect_uri: "{post_logout_redirect_uri}"
discovery.seed_hosts: [{seed_host1}, {seed_host2}, {seed_host3}]
cluster.initial_master_nodes: [{master_node1}, {master_node2}, {master_node3}]

Steps Taken:

  1. I added the client_secret to the Elasticsearch keystore using the following command:

    sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.authc.realms.oidc.customerdashboard.rp.client_secret

    The client_secret was successfully added to the keystore.

Issue:

Despite the correct configuration, Elasticsearch is unable to access the client_secret from the keystore, and I continue to receive the error:

SettingsException[The configuration setting [xpack.security.authc.realms.oidc.customerdashboard.rp.client_secret] is required]

IF I try to add client_secret as a placeholder in yml file

  1. In the elasticsearch.yml file, I have set the rp.client_secret to ${xpack.security.authc.realms.oidc.customerdashboard.rp.client_secret} to reference the keystore value.

Issue:

Exception in thread "main" java.lang.IllegalArgumentException: Circular placeholder reference 'xpack.security.authc.realms.oidc.customerdashboard.rp.client_secret' in property definitions

Additional Information:

  • I have verified the keystore entry using the following command:

    /usr/share/elasticsearch/bin/elasticsearch-keystore list

  • I have restarted Elasticsearch after making changes to the configuration.

  • The Elasticsearch version in use is 8.x (or 7.17 if applicable).

Could anyone provide insight into why Elasticsearch is unable to retrieve the client secret from the keystore and offer suggestions on how to resolve this issue?

Any help would be greatly appreciated.