SAML settings on elasticsearch and kibana

Muhammad_Umair_Baig · June 8, 2020, 10:44am

Hi Guys,
i am using kibana and elasticsearch version 7.6.2 free version and trying to enable SAML 2.0 on elasticsearch and kibana. Mentioned below are the configuration files that i am using:

elasticsearch log trace:
[node1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.security-7][0]]]).

kibana log trace:
Jun 08 10:45:47 elk kibana[158693]: {"type":"log","@timestamp":"2020-06-08T10:45:47Z","tags":["fatal","root"],"pid":158693,"message":"Error: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch\n at Object.createSecureContext (_tls_common.js:156:17)\n at Server (_tls_wrap.js:873:27)\n at new Server (https.js:62:14)\n at Object.createServer (https.js:85:10)\n at module.exports.internals.Core._createListener (/usr/share/kibana/node_modules/hapi/lib/core.js:491:79)\n at new module.exports.internals.Core (/usr/share/kibana/node_modules/hapi/lib/core.js:112:30)\n at new module.exports (/usr/share/kibana/node_modules/hapi/lib/server.js:25:18)\n at createServer (/usr/share/kibana/src/core/server/http/http_tools.js:106:18)\n at HttpServer.setup (/usr/share/kibana/src/core/server/http/http_server.js:76:48)\n at HttpService.runNotReadyServer (/usr/share/kibana/src/core/server/http/http_service.js:152:26)\n at HttpService.setup (/usr/share/kibana/src/core/server/http/http_service.js:74:18)"}

elasticsearch.yml

    node.name: node1

    network.host: 172.17.0.3

    http.port: 9200

    xpack.security.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.http.ssl.key: hwsw360.w3-969.ibm.com.key
    xpack.security.http.ssl.certificate: cert.crt
    xpack.security.transport.ssl.key: hwsw360.w3-969.ibm.com.key
    xpack.security.transport.ssl.certificate: cert.crt
    xpack.security.http.ssl.certificate_authorities: ["carootcert.der","caintermediatecert.der"]
    xpack.security.transport.ssl.certificate_authorities: ["carootcert.der","caintermediatecert.der"]
    xpack.security.http.ssl.verification_mode: none

    discovery.seed_hosts: [ "172.17.0.3" ]
    cluster.initial_master_nodes: [ "node1" ]

    path.logs: /var/log/elasticsearch
    path.data: /var/lib/elasticsearch

    xpack.security.authc.token.enabled: true
    xpack.security.authc.realms.saml.saml1:
    order: 1
    idp.metadata.path: SSO_METADATA.XML
    idp.entity_id: “MY_INTERNAL_IDP_URL”
    sp.entity_id: "https://[HOSTNAME]/"
    sp.acs: "https://[HOSTNAME]/api/security/saml/callback"
    sp.logout: "https://[HOSTNAME]/logout"

    attributes.principal: "urn:oid:0.9.2342.19200300.100.1.1"
    attributes.groups: "urn:oid:1.3.6.1.4.1.5923.1.5.1."

kibana.yml

        server.port: 5601
        server.host: "172.17.0.3"
        elasticsearch.hosts: ["https://172.17.0.3:9200"]
        elasticsearch.username: "kibana"
        elasticsearch.password: "MY_AUTO_GEN_PASSWORD"
        elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/carootcert.der", "/etc/kibana/caintermediatecert.der" ]
        logging.verbose: true
        server.ssl.enabled: true
        elasticsearch.ssl.verificationMode: none
        server.ssl.certificate: "/etc/kibana/kibana.crt"
        server.ssl.key: "/etc/kibana/hwsw360.w3-969.ibm.com.key"

ikakavas · June 8, 2020, 11:07am

Hi,

You never mentioned what your question is. Please take the time to fully structure your question with at mininum:

what you are trying to do
what you have done already
what do you expect to happen
what happens instead
what is the actual error you are getting, where and after which action

The kibana error I can see in your logs is unrelated to saml and is exactly what it says: your server.ssl.key doesn't correspond to server.ssl.certificate. In ssl, the ssl certificate binds a public key to a subject and that public key must be the from the same keypair as the private key.

magnalite · June 8, 2020, 11:14am

I'm not sure if you'll be able to do this. SSO is a feature included in the platinum license and above whereas I am assuming you are using the basic (free) license?

It would be nice if this information was included in the docs next to the sso documentation. I implemented sso only to be met with an error message saying I needed the platinum license.

Muhammad_Umair_Baig · June 8, 2020, 11:31am

Hi, Can you please share documentation link.

Muhammad_Umair_Baig · June 8, 2020, 11:59am

Hi, i have installed elastic search and kibana 7.6.2 free version on my local machine inside docker container.
i have configured saml 2.0 configuration on elasticsearch.yml file as you can see in the config file.
both elasticsearch and kibana yml files are pasted in question detail.
elasticsearch instance run fine. but when i try to run kibana service it gives me mentioned below error.

message":"Error: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch\n

ikakavas · June 8, 2020, 12:09pm

I answered to you about this in my previous message.

Also, as @magnalite told you, SAML requires a platinum (or trial) license, see : https://www.elastic.co/subscriptions

Muhammad_Umair_Baig · June 8, 2020, 1:08pm

if it is available in platinum version then its error should be different as @magnalite mentioned in his reply.

ikakavas · June 8, 2020, 1:32pm

I have explained it to you above, please read the answer. Your error is a misconfiguration in kibana that has nothing to do with SAML.

Muhammad_Umair_Baig · June 9, 2020, 2:24pm

Well i have changed both elasticsearch.yml and kibana.yml. now i am able to enable https and communication is all done on ssl:
now when i try to https://localhost:5601/ it changes the url to https://localhost:5601/api/security/saml/start?redirectURLFragment=
and gives error {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"} by the way i have configured SAML auth provider in both kibana.yml and elasticsearch.yml

Moreover i have download SSO_PROVIDER_metadata.xml from the IDP provider website. Please also confirm if i am using correct file or i need to download or generate it from somewhere?

elasticsearch.yml file configuration

xpack.license.self_generated.type: trial
xpack.security.authc.realms.saml.saml1:
      order: 0
      idp.metadata.path: SSO_PROVIDER_metadata.xml
      idp.entity_id: "https://SSO_PROVIDER_HOSTNAME/auth/sps/samlidp2/saml20"
      sp.entity_id:  "https://localhost:5601/"
      sp.acs: "https://localhost:5601/api/security/v1/saml/"
      sp.logout: "https://localhost:5601/logout"
      attributes.principal: "nameid:persistent"
      attributes.mail: "MYEMAIL@DOMAINcom"

kibana.yml file configuration

xpack.security.authProviders: [saml]
xpack.security.authc.saml.realm: saml1
server.xsrf.whitelist: [/api/security/v1/saml]

ikakavas · June 9, 2020, 2:44pm

Bottom of this page https://www.elastic.co/guide/en/elasticsearch/reference/current/trb-security-saml.html tells you how to enable logging for SAML. Once you do that, the error will be printed in your logs.

Muhammad_Umair_Baig · June 9, 2020, 3:17pm

I have updated the elasticsearch/config/log4j2.properties

logger.saml.name = org.elasticsearch.xpack.security.authc.saml
logger.saml.level = TRACE

and printed the following line in log file The 'saml' realm is not available under the current license

in order to test saml utility in trial version i have enabled mentioned below setting in elasticsearch.yml
xpack.license.self_generated.type: trial

ikakavas · June 9, 2020, 3:23pm

xpack.license.self_generated.type: trial

only works if you set it before the first time you run elasticsearch

you need Start trial API | Elasticsearch Guide [8.11] | Elastic

Muhammad_Umair_Baig · June 10, 2020, 9:02am

Thank you for the update.

Now when i hit https://localhost:5601/ url in my browser i am getting mentioned below error:
FBTSML238E The SAML message signature could not be validated.

using bin/elasticsearch-certutil utility i have generated instance.crt, ca.crt, instance.key

following is my elasticsearch.yml file configuration

xpack.security.authc.realms.saml.saml1:
  order: 0
  signing.saml_messages: ["*"]
  signing.key: certs/instance.key
  signing.certificate: certs/instance.crt
  idp.metadata.path: samlidp2_IBM_metadata_CIS_STAGE.xml
  idp.entity_id: "https://w3id.alpha.sso.ibm.com/auth/sps/samlidp2/saml20"
  sp.entity_id:  "https://localhost:5601/"
  sp.acs: "https://localhost:5601/api/security/v1/saml/"
  sp.logout: "https://localhost:5601/logout"
  attributes.principal: "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent"
  attributes.mail: "urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"

using bin/elasticsearch-saml-metadata --realm saml1 i have generated saml-elasticsearch-metadata.xml which i have uploaded to my SSO service provider.

saml-elasticsearch-metadata.xml file

<?xml version="1.0" encoding="UTF-8"?><md:EntityDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" entityID="https://localhost:5601/">
  <md:SPSSODescriptor AuthnRequestsSigned="true" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
    <md:KeyDescriptor use="signing">
      <ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
        <ds:X509Data>
          <ds:X509Certificate>MIIDIzCCAgugAwIBAgIVALMKZ64pVG7RcWXu0UbfGUDQDz9rMA0GCSqGSIb3DQEBCwUAMDQxMjAw
BgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2VuZXJhdGVkIENBMB4XDTIwMDYw
OTExMzAyNVoXDTIzMDYwOTExMzAyNVowEzERMA8GA1UEAxMIaW5zdGFuY2UwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCJoDOkA/KdLJT+l46EkJui2kIdAPs0kfBMN4pGMyix/chwIAG7
HOg5vrJKuFidhAB5xYE9rAykCky3frCNO7Pt/SVHUr0fuGCRjFlFj4gRe1jVkCkhph2sY1RFVaT3
fc09NYM7x2PH5/w7b9oNZe7fv8n9pnUO+MZ+zHlol6PlSJ9IVC5hslbHsAzuii/hSd7DXZ85inGg
lQpsEwx9qFhPpCltPQUpAX6DwujoqAnpuAiiu683w9YqdViprwqfN8mdIe0V/KTRm+NjgqPZL3BC
QSS+KI1SvhejECWYq4gjnts5aGqhTeDwU/hYRo2AcgY8U258C5yIi2pAab1WO+pnAgMBAAGjTTBL
MB0GA1UdDgQWBBTm7Ie6WN7mvq9mmN4kgYfzxOa4qDAfBgNVHSMEGDAWgBRIn/k0wKA7j8p2B4CW
2L9vyPEINDAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IBAQBkR42Q0nCQBEUfUGK+f99OBkJw
RExhkP/z/uo/cekkHvn/U7fZWLKAc+bKvLyvQP4YN4Qc9Reoe0mAE+kn3voCT2a4tRbAJ7QAdErg
Xlm/xsp32E/uU/SSq0ibP/t4B6j3jLVIzjBAclmeYt/7hnLnhbmp8HOdYFZm8wfHSKeB7ejRq/M5
+5Mkocc8jC2bNkSG5QEMtXp/N4D7lPuirpl5fscde4DkH3Ilhtv9L7eziBAicvVYV8ZIXw+4f9Dj
OnfBXhgnD3OduidXMX5U8z1dDfEC+Y//mCdenNj19ksQNIhoq1q3AxNgamJYTo06m73haI/1vh4h
LqcudEQj+RY7</ds:X509Certificate>
        </ds:X509Data>
      </ds:KeyInfo>
    </md:KeyDescriptor>
    <md:SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect" Location="https://localhost:5601/logout"/>
    <md:NameIDFormat>urn:oasis:names:tc:SAML:2.0:nameid-format:transient</md:NameIDFormat>
    <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="https://localhost:5601/api/security/v1/saml/" index="1" isDefault="true"/>
    <md:AttributeConsumingService index="1" isDefault="true">
      <md:ServiceName xml:lang="en-PK">elasticsearch</md:ServiceName>
      <md:RequestedAttribute FriendlyName="mail" Name="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"/>
      <md:RequestedAttribute FriendlyName="principal" Name="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri"/>
    </md:AttributeConsumingService>
  </md:SPSSODescriptor>
</md:EntityDescriptor>

SAML Request:

<?xml version="1.0" encoding="UTF-8"?><saml2p:AuthnRequest xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" AssertionConsumerServiceURL="https://localhost:5601/api/security/v1/saml/" Destination="https://w3id.alpha.sso.ibm.com/auth/sps/samlidp2/saml20/login" ID="_307a256d3797e51731d0861729c2c634f288f968" IssueInstant="2020-06-10T11:43:44.636Z" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Version="2.0">
  <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">https://localhost:5601/</saml2:Issuer>
  <saml2p:NameIDPolicy AllowCreate="false" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"/>
</saml2p:AuthnRequest>

ikakavas · June 10, 2020, 12:41pm

TFIM cannot verify the signature of the SAML authentication request that we are sending. We can't know why from our side, as far as we know there is no bug with our SAML request signing. You should talk to your TFIM administrator and figure out if there is more in IBM's logs or if there is a known bug and a fixpack you can apply there.

Alternatively, if this not a strict requirement, you can configure elasticsearch to not sign the SAML requests

Muhammad_Umair_Baig · June 11, 2020, 11:01am

this is strict requirement that we need to sign each saml payload. I mean every in/out should be signed.
I believe there must be signature tag that would identify that saml request is signed or not. In my request payload i dont any tag. Please confirm if the mentioned below request is signed or not.

<?xml version="1.0" encoding="UTF-8"?><saml2p:AuthnRequest xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" AssertionConsumerServiceURL="https://localhost:5601/api/security/v1/saml/" Destination="https://w3id.alpha.sso.ibm.com/auth/sps/samlidp2/saml20/login" ID="_307a256d3797e51731d0861729c2c634f288f968" IssueInstant="2020-06-10T11:43:44.636Z" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Version="2.0">
  <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">https://localhost:5601/</saml2:Issuer>
  <saml2p:NameIDPolicy AllowCreate="false" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient"/>
</saml2p:AuthnRequest>

ikakavas · June 11, 2020, 11:31am

We sign all messages when you configure signing material, so it is signed. I dont know where you copy this request from but Http-redirect binding doesn't use enveloped signatures so you shouldn't expect to see a signature element in there. Read 3.4.4.1 of https://docs.oasis-open.org/security/saml/v2.0/saml-bindings-2.0-os.pdf

Muhammad_Umair_Baig · June 11, 2020, 11:46am

the shared saml request is generated when i hit https://localhost:5601/ in the browser.
as you can see in the request payload it is binding http-post. ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
this is the error i get from sso IDP: FBTSML238E The SAML message signature could not be validated.

ikakavas · June 11, 2020, 12:34pm

Yes, I know but I shared with you documentation that explains that the signature is not enveloped in the message in this case. So the signature is calculated and becomes part of the URL. If you look at this in the logs then you might look at a point where we just print out the AuthNRequest, this is no indication that it is not signed. I think we can move forward with this at this point.

No, it's not. Please read section 3.4.1 in https://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf . The ProtocolBinding you see there doesn't dictate that this request is using that binding, but instructs the IDP that the response should use that binding.

Understood, but I have already shared with you the answer for that part:

Muhammad_Umair_Baig · June 11, 2020, 6:47pm

Thank you for your response and detailed explanation.
after successful login. i am redirected to https://localhost:5601/api/security/v1/saml/ response is {"statusCode":404,"error":"Not Found","message":"Not Found"}
May be i am missing something.

ikakavas · June 12, 2020, 5:43am

I'm pretty sure you are using documentation for setting up saml for an older version than the one you are using. Please refer back to the docs for the actual elasticsearch version you use, double check and correct the errors. If this doesn't fix your issue, paste your actual and current saml realm configuration from elasticsearch.yml and kibana.yml here , maybe someone can help you spot an error in there