ELK Stack integration with Keycloak

Hello,

We are trying to add a layer of Authorization into our ELK stack with Keycloak for our commercial product, and stumbled upon a third party plugin called Search Guard.

Although Search Guard works pretty well, after seeing the post about the news regarding this plugin, we have decided to turn away from that plugin for obvious reasons.

Now the challenge is doing the configuration manually ourselves on both Elasticsearch and Kibana (Keycloak too, of course), but unfortunately there does not seem to be a guide on this topic. I cannot imagine we are the first to try this integration, but I could not find any resources. Could you guide us on this manner?

Hi,

As far as I know keycloak supports both SAML and OpenID Connect. Depending on which you want to use, you can check our documentation at:

https://www.elastic.co/guide/en/elasticsearch/reference/7.5/saml-guide-authentication.html

and / or

https://www.elastic.co/guide/en/elasticsearch/reference/7.5/oidc-guide-authentication.html

Thanks for your answer @ikakavas

I tried to follow this guide and have this configuration for a local testing environment with Keycloak working on http://localhost:8080

xpack.security.authc.realms:
  saml.saml1:
    order: 2
    idp.metadata.path: saml/saml-elasticsearch-metadata.xml
    idp.entity_id: "http://localhost:8080"
    sp.entity_id:  "https://localhost:5601"
    sp.acs: "https://localhost:5601/api/security/v1/saml"
    sp.logout: "https://localhost:5601/logout"
    attributes.principal: "nameid:persistent"

The generated saml-elasticsearch-metadata.xml file unfortunately does not contain an entity ID

<?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://localhost:5601">
  <md:SPSSODescriptor AuthnRequestsSigned="false" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
    <md:SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://localhost:5601/logout"/>
    <md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:transient</md:NameIDFormat>
    <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://localhost:5601api/security/v1/saml" index="1" isDefault="true"/>
    <md:AttributeConsumingService index="1" isDefault="true">
      <md:ServiceName xml:lang="en-US">elasticsearch</md:ServiceName>
      <md:RequestedAttribute FriendlyName="principal" Name="nameid:persistent" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"/>
    </md:AttributeConsumingService>
  </md:SPSSODescriptor>
</md:EntityDescriptor>

And Elasticsearch throws the following ElasticsearchSecurityException:
Cannot find metadata for entity [http://localhost:8080]

Hi,

What you have generated is the Service Provider metadata but in the idp.metadata.path you need to set the Identity Provider metadata. In this case KeyCloak is the Identity Provider so you need to configure KeyCloak and get its metadata.

This is meant to be an introductory blog post on how SAML works and not a complete guide. This is why I shared https://www.elastic.co/guide/en/elasticsearch/reference/7.5/saml-guide-authentication.html with you above.

My bad! Thanks for your reply.

I managed to enable SSL communication between Elasticsearch and Kibana, so that the login page comes up on https://localhost:5601 but it is the one from Kibana, and not from Keycloak. What could be the problem?

Apologies, but I'm not really sure how this is relevant to what we are discussing here. Can you clarify ?

Kibana needs configuration too. I should have pointed you to https://www.elastic.co/guide/en/elasticsearch/reference/7.5/saml-guide.html ( instead of https://www.elastic.co/guide/en/elasticsearch/reference/7.5/saml-guide-authentication.html), see all the sections here and also https://www.elastic.co/guide/en/elasticsearch/reference/7.5/saml-kibana.html

Second paragraph from your link states

In particular, since your Elasticsearch nodes have been configured to use TLS on the HTTP interface, you must configure Kibana to use a https URL to connect to Elasticsearch, and you may need to configure elasticsearch.ssl.certificateAuthorities to trust the certificates that Elasticsearch has been configured to use.

So configuring TLS is relevant, am I mistaken?

After adding the following to kibana.yml

xpack.security.authc.providers: [saml]
xpack.security.authc.saml.realm: saml1
server.xsrf.whitelist: [/api/security/v1/saml]

I received this error:
[authentication][plugins][security] Authentication attempt failed: [security_exception] current license is non-compliant for [saml], with { license.expired.feature="saml" }

Is this because SAML is only supported in the platinum license, or is there anything else to do?

Yes, but it was not clear why you thought that configuring it would cause the problem you mentioned after that.

Yeah, saml is available on platinum only but you can try it out with a trial license for 30 days If you plan on getting a licence in the future.

Thanks, I have just activated a trial license to see if it is indeed the solution for us.

Unfortunately I receive the following error:

log   [07:22:30.873] [error][admin][elasticsearch] Request error, retrying
HEAD https://localhost:9200/.apm-agent-configuration => write EPROTO 17540:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:openssl\ssl\record\ssl3_record.c:252:

log   [07:22:30.879] [error][admin][elasticsearch] Request error, retrying
GET https://localhost:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => write EPROTO 17540:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:openssl\ssl\record\ssl3_record.c:252:

log   [07:22:30.911] [warning][admin][elasticsearch] Unable to revive connection: https://localhost:9200/
log   [07:22:30.914] [warning][admin][elasticsearch] No living connections

Then I would assume that activating the trial license is not the only thing you did :slight_smile: . This error message indicates that elasticsearch is not setup for TLS on the HTTP layer. Can you please share your actual current configuration from elasticsearch.yml and kibana.yml ( <- This is a nice thing to do whenever changing/updating something as it allows people that participate in these forums to get back to you quicker and with actual suggestions on what might be wrong )

1 Like

I had to disable X-Pack because of the license issue :slight_smile: I don't quite understand why that had to be done, but Elasticsearch specifically tells that you should disable X-Pack if you want to have SAML integration.

elasticsearch.yml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: elastic-stack-ca.p12

xpack.security.authc.token.enabled: true

xpack.security.authc.realms:
  saml.saml1:
    order: 2
    idp.metadata.path: saml/keycloak-descriptor.xml
    idp.entity_id: "http://localhost:8080/auth/realms/demo"
    sp.entity_id:  "https://localhost:5601"
    sp.acs: "https://localhost:5601/api/security/v1/saml"
    sp.logout: "https://localhost:5601/logout"
    attributes.principal: "nameid:persistent"

kibana.yml

xpack.security.authc.providers: [saml]
xpack.security.authc.saml.realm: saml1
server.xsrf.whitelist: [/api/security/v1/saml]

xpack.security.enabled: true

Once I go over to https://localhost:5601 now, I see an error coming from Keycloak, which is actually a good sign :smiley:
# We are sorry...
Unknown login requester

I decided to switch over to OIDC and found that actually way easier.

I have followed this guide, set up following configurations:

elasticsearch.yml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: elastic-stack-ca.p12

xpack.security.authc.token.enabled: true

xpack.security.authc.realms:
  oidc.oidc1:
    order: 2
    rp.client_id: "kibana"
    rp.response_type: code
    rp.redirect_uri: "https://localhost:5601/api/security/v1/oidc"
    op.issuer: "http://localhost:8080/auth/realms/demo"
    op.authorization_endpoint: "http://localhost:8080/auth/realms/demo/protocol/openid-connect/auth"
    op.token_endpoint: "http://localhost:8080/auth/realms/demo/protocol/openid-connect/token"
    op.jwkset_path: certs.json
    op.userinfo_endpoint: "http://localhost:8080/auth/realms/demo/protocol/openid-connect/userinfo"
    op.endsession_endpoint: "http://localhost:8080/auth/realms/demo/protocol/openid-connect/logout"
    rp.post_logout_redirect_uri: "https://localhost:5601/logged_out"
    claims.principal: email

kibana.yml

xpack.security.authc.providers: [oidc]
xpack.security.authc.oidc.realm: "oidc1"
server.xsrf.whitelist: [/api/security/v1/oidc]

xpack.security.enabled: true

Now I get the login screen from Keycloak once I go over to https://localhost:5601. Unfortunately, once I try to login with one of the users I have configured in Keycloak, the following error message is thrown:
Authentication to realm oidc1 failed - Failed to authenticate user with OpenID Connect (Caused by ElasticsearchSecurityException[Failed to parse or validate the ID Token]; nested: BadJOSEException[Signed JWT rejected: Another algorithm expected, or no matching key(s) found];)

Could you help me get over this issue?

The algorithm stated in my certs.json is RS256, I fetched this data from Keycloak's OpenID configuration under http://localhost:8080/auth/realms/demo/.well-known/openid-configuration.

What exactly made it difficult for SAML ? As far as I can see from the error message, you hadn't configured SAML in Keycloak yet. Other than that, was there something on the Elastic Stack side you found difficult/confusing ?

The default value for the Elastic Stack RP configuration is also RS256 .

Is it possible that Keycloak has different configuration ( different keys ) for different realms and you haven't configured the one called demo , but using a different one ?

You can enable trace level logging in Elasticsearch with

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security.authc.oidc": "trace"
  }
}

or alternatively by adding the following lines to the end of the log4j2.properties configuration file in the ES_PATH_CONF:

logger.oidc.name = org.elasticsearch.xpack.security.authc.oidc
logger.oidc.level = TRACE

and see if we get any more detail there

I kept receiving the same error message "Unknown login requester" from Keycloak with SAML. I was probably missing some values somewhere. I am not a systems engineer, so unfortunately I cannot tell you what went wrong.

Thanks for trace level logging information. I see the following after I click

[2019-12-12T11:21:30,119][TRACE][o.e.x.s.a.o.OpenIdConnectAuthenticator] [S2MW31] OpenID Connect Provider redirected user to [/api/security/v1/oidc?state=iaAXRLJaJq_nnfc_ujdlNOHseOWJHZw3pqpeJ2Cnf2g&session_state=f81d040e-ec66-44bd-958f-2be6c6aad4e1&code=e645dbc2-b415-48a8-9bab-19b115140c72.f81d040e-ec66-44bd-958f-2be6c6aad4e1.d999e26b-a38b-482c-bcf8-e7e3eae74f43]. Expected Nonce is [GRxwUBq9b-rxe1hA7XzCQOV2lj_yvKM3AhwWiMOF99k] and expected State is [iaAXRLJaJq_nnfc_ujdlNOHseOWJHZw3pqpeJ2Cnf2g]
[2019-12-12T11:21:30,342][TRACE][o.e.x.s.a.o.OpenIdConnectAuthenticator] [S2MW31] Received Token Response from OP with status [200] and content [{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIya3pRd3dvTDd1ZUIwcGxiRVdtU2ZTbzI0SzVhcHFjNDNIbktZc2hVTm5RIn0.eyJqdGkiOiI4YzA3OWYyNC1mNmFmLTQwNmQtOWRlNi05MDFmMmRmYmY5NDEiLCJleHAiOjE1NzYxNDYzOTAsIm5iZiI6MCwiaWF0IjoxNTc2MTQ2MDkwLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvYXV0aC9yZWFsbXMvZGVtbyIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiIzZmE5YjUwZC05M2M4LTRjYTktOGE4MC1lOTJhMzIxYjFiZjUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJraWJhbmEiLCJub25jZSI6IkdSeHdVQnE5Yi1yeGUxaEE3WHpDUU9WMmxqX3l2S00zQWh3V2lNT0Y5OWsiLCJhdXRoX3RpbWUiOjE1NzYxNDYwOTAsInNlc3Npb25fc3RhdGUiOiJmODFkMDQwZS1lYzY2LTQ0YmQtOTU4Zi0yYmU2YzZhYWQ0ZTEiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbImh0dHBzOi8vbG9jYWxob3N0OjU2MDEiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIkRhc2hib2FyZCBVc2VyIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJlbWFpbF92ZXJpZmllZCI6WyJEYXNoYm9hcmQgVXNlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXSwibmFtZSI6InRlc3QxIHVzZXIxIiwicHJlZmVycmVkX3VzZXJuYW1lIjoidGVzdHVzZXIxIiwiZ2l2ZW5fbmFtZSI6InRlc3QxIiwiZmFtaWx5X25hbWUiOiJ1c2VyMSIsImVtYWlsIjoidGVzdEB0ZXN0LmNvbSJ9.Up4T2J6bzL558gsdjDJGU_3Fj_2g5hKTH2wkzfuNEfZhOKTSgSPwx1tp-dn2Ic6sUpTQl9m_YInVZtxuaNSj54_My8QgEGUTBfFTbSKa_WPBf8o6WRNhWTxwH87zVdOsfhU_DoPinRLiqyyvaxDfL0wxZ2lfowiLv3-TtM5JG8BTN2kqttI4RZenAV69dF4LkL8l9Grg89sQ2w8f-HkMMd2UwW3mxxWtqIMKvhyK1_j-7PGFiLrc3ti66OnfPZZU5qynVW4yvugwY6M_8ZHVP-fr1Arbp5BuUMxYR_fkVDMcAG64ObB4EQuZ8Py-_exaTLZvDK3I191ChqhFxGUlvw","expires_in":300,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI5MWM1ZjE1OS1lOTllLTQ3MjAtYWM2YS1mZDVjYWE4OGNmOTcifQ.eyJqdGkiOiIyNjg4ZWQzZi1jOTczLTQ3OTItYmQyNi1hZTQ1ZWI2OTQyMjAiLCJleHAiOjE1NzYxNDc4OTAsIm5iZiI6MCwiaWF0IjoxNTc2MTQ2MDkwLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvYXV0aC9yZWFsbXMvZGVtbyIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MC9hdXRoL3JlYWxtcy9kZW1vIiwic3ViIjoiM2ZhOWI1MGQtOTNjOC00Y2E5LThhODAtZTkyYTMyMWIxYmY1IiwidHlwIjoiUmVmcmVzaCIsImF6cCI6ImtpYmFuYSIsIm5vbmNlIjoiR1J4d1VCcTliLXJ4ZTFoQTdYekNRT1YybGpfeXZLTTNBaHdXaU1PRjk5ayIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6ImY4MWQwNDBlLWVjNjYtNDRiZC05NThmLTJiZTZjNmFhZDRlMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJEYXNoYm9hcmQgVXNlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIn0.onjK5H901YA2gCsgZbRf_4LSjiaJYkITwfybQENCGL0","token_type":"bearer","id_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIya3pRd3dvTDd1ZUIwcGxiRVdtU2ZTbzI0SzVhcHFjNDNIbktZc2hVTm5RIn0.eyJqdGkiOiJmNDAyYmM0Mi1lNWFjLTQwZTQtOWM4MS1kNjY2ZDczM2E1YWYiLCJleHAiOjE1NzYxNDYzOTAsIm5iZiI6MCwiaWF0IjoxNTc2MTQ2MDkwLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvYXV0aC9yZWFsbXMvZGVtbyIsImF1ZCI6ImtpYmFuYSIsInN1YiI6IjNmYTliNTBkLTkzYzgtNGNhOS04YTgwLWU5MmEzMjFiMWJmNSIsInR5cCI6IklEIiwiYXpwIjoia2liYW5hIiwibm9uY2UiOiJHUnh3VUJxOWItcnhlMWhBN1h6Q1FPVjJsal95dktNM0Fod1dpTU9GOTlrIiwiYXV0aF90aW1lIjoxNTc2MTQ2MDkwLCJzZXNzaW9uX3N0YXRlIjoiZjgxZDA0MGUtZWM2Ni00NGJkLTk1OGYtMmJlNmM2YWFkNGUxIiwiYWNyIjoiMSIsImVtYWlsX3ZlcmlmaWVkIjpbIkRhc2hib2FyZCBVc2VyIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiJdLCJuYW1lIjoidGVzdDEgdXNlcjEiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJ0ZXN0dXNlcjEiLCJnaXZlbl9uYW1lIjoidGVzdDEiLCJmYW1pbHlfbmFtZSI6InVzZXIxIiwiZW1haWwiOiJ0ZXN0QHRlc3QuY29tIn0.QGxOPzGI-DPFOnO8hvek0-I0AEkULH4Bk3bMNHCFMfZnve-FZk7w2cSagvnL-xmvTWMxniNuxJlEudFcHFhcKmcusWYE24MJqIt0NPNO1OerHOoBjQbAOzf1eSWvMyLtFnFOoUROp8s_8bdWlIJbHTmu1chNcRZDcbNTgOlllvP-Mvtd4OGYl9roj_b8VN4LVG2Cn5c_JX6R0UMifM9rDhQ0bUf4PpHxLXI0Jr0TX6A79J3NLxdCpY_k_d7nBoD57RT7ekzf4qq1beOyKXlyFAvFqPhS-kFm9P68-K9NlqMRSqI7UzFIyGjxxL4pcfQZcsBttqwRpO-DTtJiWNHv4g","not-before-policy":0,"session_state":"f81d040e-ec66-44bd-958f-2be6c6aad4e1","scope":"openid email profile"}] 
[2019-12-12T11:21:30,363][TRACE][o.e.x.s.a.o.OpenIdConnectAuthenticator] [S2MW31] Successfully exchanged code for ID Token: [com.nimbusds.jwt.SignedJWT@6bce1570] and Access Token [eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIya3pRd3dvTDd1ZUIwcGxiRVdtU2ZTbzI0SzVhcHFjNDNIbktZc2hVTm5RIn0.eyJqdGkiOiI4YzA3OWYyNC1mNmFmLTQwNmQtOWRlNi05MDFmMmRmYmY5NDEiLCJleHAiOjE1NzYxNDYzOTAsIm5iZiI6MCwiaWF0IjoxNTc2MTQ2MDkwLCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvYXV0aC9yZWFsbXMvZGVtbyIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiIzZmE5YjUwZC05M2M4LTRjYTktOGE4MC1lOTJhMzIxYjFiZjUiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJraWJhbmEiLCJub25jZSI6IkdSeHdVQnE5Yi1yeGUxaEE3WHpDUU9WMmxqX3l2S00zQWh3V2lNT0Y5OWsiLCJhdXRoX3RpbWUiOjE1NzYxNDYwOTAsInNlc3Npb25fc3RhdGUiOiJmODFkMDQwZS1lYzY2LTQ0YmQtOTU4Zi0yYmU2YzZhYWQ0ZTEiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbImh0dHBzOi8vbG9jYWxob3N0OjU2MDEiXSwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbIkRhc2hib2FyZCBVc2VyIiwib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImFjY291bnQiOnsicm9sZXMiOlsibWFuYWdlLWFjY291bnQiLCJtYW5hZ2UtYWNjb3VudC1saW5rcyIsInZpZXctcHJvZmlsZSJdfX0sInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJlbWFpbF92ZXJpZmllZCI6WyJEYXNoYm9hcmQgVXNlciIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXSwibmFtZSI6InRlc3QxIHVzZXIxIiwicHJlZmVycmVkX3VzZXJuYW1lIjoidGVzdHVzZXIxIiwiZ2l2ZW5fbmFtZSI6InRlc3QxIiwiZmFtaWx5X25hbWUiOiJ1c2VyMSIsImVtYWlsIjoidGVzdEB0ZXN0LmNvbSJ9.Up4T2J6bzL558gsdjDJGU_3Fj_2g5hKTH2wkzfuNEfZhOKTSgSPwx1tp-dn2Ic6sUpTQl9m_YInVZtxuaNSj54_My8QgEGUTBfFTbSKa_WPBf8o6WRNhWTxwH87zVdOsfhU_DoPinRLiqyyvaxDfL0wxZ2lfowiLv3-TtM5JG8BTN2kqttI4RZenAV69dF4LkL8l9Grg89sQ2w8f-HkMMd2UwW3mxxWtqIMKvhyK1_j-7PGFiLrc3ti66OnfPZZU5qynVW4yvugwY6M_8ZHVP-fr1Arbp5BuUMxYR_fkVDMcAG64ObB4EQuZ8Py-_exaTLZvDK3I191ChqhFxGUlvw]
[2019-12-12T11:21:30,383][DEBUG][o.e.x.s.a.o.OpenIdConnectRealm] [S2MW31] Failed to consume the OpenIdConnectToken 
org.elasticsearch.ElasticsearchSecurityException: Failed to parse or validate the ID Token
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.getUserClaims(OpenIdConnectAuthenticator.java:259) [x-pack-security-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.lambda$authenticate$0(OpenIdConnectAuthenticator.java:193) [x-pack-security-7.4.0.jar:7.4.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.handleTokenResponse(OpenIdConnectAuthenticator.java:542) [x-pack-security-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.access$600(OpenIdConnectAuthenticator.java:125) [x-pack-security-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator$2.completed(OpenIdConnectAuthenticator.java:479) [x-pack-security-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator$2.completed(OpenIdConnectAuthenticator.java:476) [x-pack-security-7.4.0.jar:7.4.0]
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) [httpcore-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) [httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.11.jar:4.4.11]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) [httpcore-nio-4.4.11.jar:4.4.11]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: com.nimbusds.jose.proc.BadJOSEException: Signed JWT rejected: Another algorithm expected, or no matching key(s) found
	at com.nimbusds.jwt.proc.DefaultJWTProcessor.<clinit>(DefaultJWTProcessor.java:100) ~[?:?]
	at com.nimbusds.openid.connect.sdk.validators.IDTokenValidator.validate(IDTokenValidator.java:285) ~[?:?]
	at com.nimbusds.openid.connect.sdk.validators.IDTokenValidator.validate(IDTokenValidator.java:224) ~[?:?]
	at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.getUserClaims(OpenIdConnectAuthenticator.java:226) ~[?:?]
	... 21 more

Is it possible that Keycloak has different configuration ( different keys ) for different realms and you haven't configured the one called demo , but using a different one ?

You are actually right. For some reason, it was wrongly configured. I switched the necessary endpoints, and added my client secret for realm ROOT and it worked. No idea why.

Everything is working now. I will just document it here in case someone else needs this information, because it was a somewhat painful process for me. I assume I was doing something in a weird/wrong way, and I am not suggesting this is the way to do it. (You should not do this on production anyways)

elasticsearch.yml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: elastic-stack-ca.p12
xpack.security.http.ssl.truststore.path: elastic-stack-ca.p12

xpack.security.authc.token.enabled: true

xpack.security.authc.realms:
  oidc.oidc1:
    order: 2
    rp.client_id: "kibana"
    rp.response_type: code
    rp.redirect_uri: "https://localhost:5601/api/security/v1/oidc"
    op.issuer: "http://localhost:8080/auth/realms/master"
    op.authorization_endpoint: "http://localhost:8080/auth/realms/master/protocol/openid-connect/auth"
    op.token_endpoint: "http://localhost:8080/auth/realms/master/protocol/openid-connect/token"
    op.jwkset_path: certs.json
    op.userinfo_endpoint: "http://localhost:8080/auth/realms/master/protocol/openid-connect/userinfo"
    op.endsession_endpoint: "http://localhost:8080/auth/realms/master/protocol/openid-connect/logout"
    rp.post_logout_redirect_uri: "https://localhost:5601/logged_out"
    claims.principal: preferred_username

You need to find these values from the "well known URL" of your IdP, for a Keycloak server running locally this will be http://localhost:8080/auth/realms/master/.well-known/openid-configuration.

kibana.yml

xpack.security.authc.providers: [oidc]
xpack.security.authc.oidc.realm: "oidc1"
server.xsrf.whitelist: [/api/security/v1/oidc]

xpack.security.enabled: true

You also need to change elasticsearch.hosts to https, set elasticsearch.username and elasticsearch.password, and server.ssl.enabled, server.ssl.certificate, server.ssl.key fields.

You must enable the trial license in order to do all this, because these features are in the platinum license. https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trial.html

You need to add the client secret from Keycloak to Elasticsearch using the keystore tool.
elasticsearch-keystore add xpack.security.authc.realms.oidc.oidc1.rp.client_secret is the command.

Then you need to create roles and role mappings on Elasticsearch, also create roles and users on Keycloak, so that you can actually use these to log into Kibana.

And that is it. Elasticsearch guides are actually enough on their own, but for a newbie like me it was still somewhat confusing, this could just be me.

Thanks @ikakavas for your help!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.