# Elasticsearch SSL/TLS PKI realm setup

**URL:** <https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [March 1, 2021, 11:58am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790 "2021-03-01T11:58:33Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 1, 2021, 11:58am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/1 "2021-03-01T11:58:33Z")

</div>

Hello,  
I am trying to setup PKI realm on my onpremise ELK stack on a k8s cluster and I am using helm as the package manager  
Elasticsearch version: 7.11.1

I have created an organization signed server certificate, using which I have enabled xpack security at transport and http layer. The certificate is password protected, so keystore values are defined as well for the following

```
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
xpack.security.http.ssl.keystore.secure_password
xpack.security.http.ssl.truststore.secure_password

```

Here is the helm values.yaml file

```
  elasticsearch.yml: |

   xpack.security.enabled: true

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.verification_mode: certificate

   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.pem

   xpack.security.http.ssl.enabled: true

   xpack.security.http.ssl.verification_mode: certificate

   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.pem

   xpack.security.http.ssl.client_authentication: optional

   xpack.security.authc.realms.native.local.order: 0

   xpack.security.authc.realms.active_directory.my_ad.order: 1

   xpack.security.authc.realms.active_directory.my_ad.domain_name: myad

   xpack.security.authc.realms.active_directory.my_ad.url: ldap://myldap:389

   xpack.security.authc.realms.pki.pki1.order: 2

   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca.pem']

```

and created a role mapping in kibana

 ![role_mappping](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6c2fc0d688b2f634acd6dc6471ab074f00e2a5f0.png)

After that I created another certificate to be used by the client, signed by organization (same who signed the server certificates)

The ELK stack itself is working fine and i am able to login using as AD user, built-in user. However when i am trying to use the client certificate, it fails to authenticate. Here i am calling the authentication api from postman

 ![postman](https://us1.discourse-cdn.com/elastic/original/3X/d/4/d46ae2d031ac114df92f9b09cdba0216a6afb014.png)  
error details

```
                "reason": "missing authentication credentials for REST request [/_xpack/security/_authenticate?pretty]",

                "header": {

                    "WWW-Authenticate": [

                        "Basic realm=\"security\" charset=\"UTF-8\"",

                        "Bearer realm=\"security\"",

                        "ApiKey"

                    ]

                }

```

here is a curl output

 ![curl1](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c28d1f9ffdd088737ef350eb687aa9b24e507518.png)  
 ![curl2](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2d363eb2451c98aa99b0f6ec0ee0515e0fb16e52.png)  
Really appreciate any pointers.

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [March 1, 2021, 1:39pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/2 "2021-03-01T13:39:29Z")

</div>

Your curl reproduction contains `--cacert` twice, the first one (pointing to `client.cer`) needs to be `--cert` instead

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 1, 2021, 2:58pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/3 "2021-03-01T14:58:46Z")

</div>

Thanks @ikakavas

Couldnt get curl to work. While i am still at it - Do you see any issues with the setup and/or the error in postman ?

 ![curl3](https://us1.discourse-cdn.com/elastic/original/3X/3/f/3f5d1e4efdb8a1febf2f8c07790e7eef8eabac30.png)

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 1, 2021, 3:47pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/4 "2021-03-01T15:47:35Z")

</div>

Not sure if i am doing it right, but here it is. getting the same error like i saw in postman  
**`WWW-Authenticate`** shy is it not showing pki realm. do you see any issues in my setup ?

 ![curl4](https://us1.discourse-cdn.com/elastic/original/3X/b/c/bc6e70fb304b2c0b06de622282865f5f23c8acbd.png)

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 2, 2021, 3:06am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/5 "2021-03-02T03:06:21Z")

</div>

The most likely explanation is that the server cannot establish is not a trust path between `client.crt` and `ca.pem`

You have a couple of options to try and debug that:

1. Use external tools such as `openssl verify` to check that the client certificate verifies against the configured certificate authority
2. Switch `xpack.security.http.ssl.client_authentication` to `required` and check the ES log for errors. Debugging optional client auth is hard because there is no error when it fails.

Also, your curl commands are all passing `--cacert` as well as `-k`, which is telling curl to verify the server against `ca.cer`, but also don't verify it.  
It would be better for you to drop the `-k` and get it working with `--cacert` so that you know you are verifying the correct behaviour.

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 2, 2021, 12:38pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/6 "2021-03-02T12:38:05Z")

</div>

After setting client\_authentication to **required** , I get error within elasticsearch node

```
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
"at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]",
"at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]",
"at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"... 16 more"] }

```

Was able to verify the client certificate

 ![curl_working](https://us1.discourse-cdn.com/elastic/original/3X/8/0/80b2b0d33dab23c65e74641ba500ff0153983aa1.png)  
here is the changed yaml (added root, intermediate ca to certificate authorities list)

```
elasticsearch.yml: |

   xpack.security.enabled: true

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.verification_mode: certificate

   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.transport.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']

   xpack.security.http.ssl.enabled: true

   xpack.security.http.ssl.verification_mode: certificate

   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']

   xpack.security.http.ssl.client_authentication: required

   xpack.security.authc.realms.native.local.order: 0

   xpack.security.authc.realms.active_directory.my_ad.order: 1

   xpack.security.authc.realms.active_directory.my_ad.domain_name: adep01.nordic.rsa-ins.com

   xpack.security.authc.realms.active_directory.my_ad.url: ldap://ADEP01.nordic.rsa-ins.com:389

   xpack.security.authc.realms.pki.pki1.order: 2

   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']
```

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 3, 2021, 1:50am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/7 "2021-03-03T01:50:40Z")

</div>

> [@anoopkv](#):
>
> ```auto
> xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']
> 
> ```

I think this is your issue.

You are listing each element in the signing chain as a separate item in the `certificate_authorities` array, but that's not how it works.

In fact, each file that is listed in `certificate_authorities` is a supposed to be a chain in itself. The array structure allows you to trust multiple CA chains, not to specify each certificate individually.

Instead you need to merge those certificates into a single PEM cert chain file and list that.

```auto
cat /usr/share/elasticsearch/config/certs/intermediate-ca.pem \
    /usr/share/elasticsearch/config/certs/root-ca.pem \
  > /usr/share/elasticsearch/config/certs/ca-chain.pem

```

```auto
xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

```

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 3, 2021, 6:10am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/8 "2021-03-03T06:10:20Z")

</div>

If you see my initial setup, I had only kept one certificate - root ca.  
Anyways I have tried your suggestion

`type D:\Certificates\elk\intermediate-ca.pem D:\Certificates\elk\root-ca.pem > D:\Certificates\elk\ca-chain.pem`

```
elasticsearch.yml: |

   xpack.security.enabled: true

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.verification_mode: certificate

   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.transport.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

   xpack.security.http.ssl.enabled: true

   xpack.security.http.ssl.verification_mode: certificate

   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

   xpack.security.http.ssl.client_authentication: required

   xpack.security.authc.realms.native.local.order: 0

   xpack.security.authc.realms.active_directory.my_ad.order: 1

   xpack.security.authc.realms.active_directory.my_ad.domain_name: adep01.nordic.rsa-ins.com

   xpack.security.authc.realms.active_directory.my_ad.url: ldap://ADEP01.nordic.rsa-ins.com:389

   xpack.security.authc.realms.pki.pki1.order: 2

   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

```

And i get the exact same error as last time ( i am not pasting the full error again, as it is same as earlier)  
`"Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain"`

 ![curl_cat_cas](https://us1.discourse-cdn.com/elastic/original/3X/2/3/231c213653ff63d57e9c397e37a4127af8cdcaec.png)

Not sure if it is of any info, when I try to run openssl verify, I get error

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 6, 2021, 2:06pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/9 "2021-03-06T14:06:26Z")

</div>

Hi @TimV

Just for test I am trying to use the same server certificate.  
I am able to verify it using openssl

 ![openssl_verify](https://us1.discourse-cdn.com/elastic/original/3X/9/8/98ca9510f859ebe54871ca777177bd328e0fe648.png)

Using that certificate and key, I try to do a curl and get error for "missing authentication credentials"

 ![curl_client_cert_auth](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a814d8304548addb2f6b1207a96f0edb2250a31.png)  
my current config values.

```
  elasticsearch.yml: |
   xpack.security.enabled: true
   xpack.security.transport.ssl.enabled: true
   xpack.security.transport.ssl.verification_mode: certificate
   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate.p12
   xpack.security.transport.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']
   xpack.security.http.ssl.enabled: true
   xpack.security.http.ssl.verification_mode: certificate
   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate.p12
   xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']
   xpack.security.http.ssl.client_authentication: optional
   xpack.security.authc.realms.native.local.order: 0
   xpack.security.authc.realms.active_directory.my_ad.order: 1
   xpack.security.authc.realms.active_directory.my_ad.domain_name: adep01.nordic.rsa-ins.com
   xpack.security.authc.realms.active_directory.my_ad.url: ldap://myldap:389
   xpack.security.authc.realms.pki.pki1.order: 2
   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

```

1. Why does my cluster fail, when i make client\_authentication "required". Only elastic was running in my cluster
2. Is my realm setup correct?

I am stuck with issue for more than two weeks now ☹ . Any idea what could be wrong ?

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 7, 2021, 2:22am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/10 "2021-03-07T02:22:08Z")

</div>

Your curl output (which I can't quote here because you pasted a screenshot - please don't do that, you should copy the text directly) says it is using HTTP/2

Elasticsearch doesn't support HTTP/2, so if the curl output is to be believed, then whatever you are connecting too isn't an Elasticsearch node. Do you have a proxy or load balancer in place?

> [@anoopkv](#):
>
> I am stuck with issue for more than two weeks now ☹ .

If you have a paid license then you should contact support, they can give you far more effective support than we can.

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 7, 2021, 3:05pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/11 "2021-03-07T15:05:29Z")

</div>

Hi @TimV

Thanks a lot for that pointer, you are spot on. My entire ELK stack is setup on a k8s cluster, and I am using an nginx controller as the ingress

When I enable port-forwarding and do a curl. it seems to work fine  
So afterall the PKI setup, the client and server certificates all are correct  
C:\Program Files\Git\usr\bin\>curl.exe -v -key C:\cert\elk\_key.pem -cert C:\cert\elk\_certificate.crt [https://127.0.0.1:9200/\_xpack/security/\_authenticate](https://127.0.0.1:9200/_xpack/security/_authenticate)  
\* Rebuilt URL to: C:\cert\elk\_key.pem/  
\* Port number ended with ''  
\* Closing connection -1  
curl: (3) Port number ended with ''  
\* Rebuilt URL to: C:\cert\elk\_certificate.crt/  
\* Port number ended with ''  
\* Closing connection -1  
curl: (3) Port number ended with ''  
\* Trying 127.0.0.1...  
\* TCP\_NODELAY set  
\* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)  
\* schannel: disabled server certificate revocation checks  
\* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.  
\* schannel: using IP address, SNI is not supported by OS.  
\* schannel: sending initial handshake data: sending 147 bytes...  
\* schannel: sent initial handshake data: sent 147 bytes  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: failed to receive handshake, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 4096  
\* schannel: encrypted data buffer: offset 4096 length 4096  
\* schannel: received incomplete message, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 1024  
\* schannel: encrypted data buffer: offset 5120 length 5120  
\* schannel: received incomplete message, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 1024  
\* schannel: encrypted data buffer: offset 6144 length 6144  
\* schannel: received incomplete message, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 62  
\* schannel: encrypted data buffer: offset 6206 length 7168  
\* schannel: sending next handshake data: sending 4649 bytes...  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 7168  
\* schannel: encrypted data buffer: offset 7168 length 7168  
\* schannel: received incomplete message, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 1024  
\* schannel: encrypted data buffer: offset 8192 length 8192  
\* schannel: received incomplete message, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 1024  
\* schannel: encrypted data buffer: offset 9216 length 9216  
\* schannel: received incomplete message, need more data  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)  
\* schannel: encrypted data got 945  
\* schannel: encrypted data buffer: offset 10161 length 10240  
\* schannel: SSL/TLS handshake complete  
\* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)  
\* schannel: stored credential handle in session cache  
\> GET /\_xpack/security/\_authenticate HTTP/1.1  
\> Host: 127.0.0.1:9200  
\> User-Agent: curl/7.55.1  
\> Accept: _/_  
\> Referer: y  
\>  
\* schannel: client wants to read 102400 bytes  
\* schannel: encdata\_buffer resized 103424  
\* schannel: encrypted data buffer: offset 0 length 103424  
\* schannel: encrypted data got 523  
\* schannel: encrypted data buffer: offset 523 length 103424  
\* schannel: decrypted data length: 494  
\* schannel: decrypted data added: 494  
\* schannel: decrypted data cached: offset 494 length 102400  
\* schannel: encrypted data buffer: offset 0 length 103424  
\* schannel: decrypted data buffer: offset 494 length 102400  
\* schannel: schannel\_recv cleanup  
\* schannel: decrypted data returned 494  
\* schannel: decrypted data buffer: offset 0 length 102400  
\< HTTP/1.1 200 OK  
\< Warning: 299 Elasticsearch-7.11.1-ff17057114c2199c9c1bbecc727003a907c0db7a "[GET /\_xpack/security/\_authenticate] is deprecated! Use [GET /\_security/\_authenticate] instead."  
\< content-type: application/json; charset=UTF-8  
\< content-length: 233  
\<  
{"username":"uid1","roles":,"full\_name":null,"email":null,"metadata":{"pki\_dn":"CN=uid1"},"enabled":true,"authentication\_realm":{"name":"pki1","type":"pki"},"lookup\_realm":{"name":"pki1","type":"pki"},"authentication\_type":"realm"}\* Connection #0 to host 127.0.0.1 left intact

When i curl it directly it is not working

```
C:\Program Files\Git\usr\bin>curl.exe -v -key C:\cert\elk_key.pem -cert C:\cert\elk_certificate.crt https://elasticsearch.mydomain.com/_xpack/security/_authenticate
* Rebuilt URL to: C:\cert\elk_key.pem/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
* Rebuilt URL to: C:\cert\elk_certificate.crt/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
* Trying 10.204.169.6...
* Trying 10.204.169.6...
* TCP_NODELAY set
* Connected to elasticsearch.mydomain.com (10.204.169.6) port 443 (#0)
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: sending initial handshake data: sending 195 bytes...
* schannel: sent initial handshake data: sent 195 bytes
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 2/3)
* schannel: encrypted data got 2752
* schannel: encrypted data buffer: offset 2752 length 4096
* schannel: sending next handshake data: sending 4649 bytes...
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /_xpack/security/_authenticate HTTP/1.1
> Host: elasticsearch.mydomain.com
> User-Agent: curl/7.55.1
> Accept: */*
> Referer: y
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 895
* schannel: encrypted data buffer: offset 895 length 103424
* schannel: decrypted data length: 866
* schannel: decrypted data added: 866
* schannel: decrypted data cached: offset 866 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 866 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 866
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401 Unauthorized
< Date: Sun, 07 Mar 2021 09:29:26 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 517
< Connection: keep-alive
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}* Connection #0 to host elasticsearch.mydomain.com left intact

```

Tried the setup suggested, but does not seem to make any difference.  
[nginx controller setup](https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/ingress.yaml) May be its not the right forum, but Can you please take a look at my setup

```
ingress:
  enabled: true
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret2"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
  path: /
  hosts:
    - elasticsearch.mydomain.com
  tls:
    - secretName: elastic-tls
      hosts:
        - elasticsearch.mydomain.com
  rules:
  - host: elasticsearch.mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: elasticsearch-master
          servicePort: 9200

```

> [@TimV](#):
>
> If you have a paid license then you should contact support, they can give you far more effective support than we can.

I am in catch22 situation, i need to complete a POC and evaluate it before my org can buy the license ☹

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 7, 2021, 11:34pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/12 "2021-03-07T23:34:52Z")

</div>

> [@anoopkv](#):
>
> My entire ELK stack is setup on a k8s cluster, and I am using an nginx controller as the ingress

You cannot perform PKI authentication through an HTTP layer proxy. In your case nginx is terminating TLS, so Elasticsearch never sees the client certificate. That means it cannot perform user authentication based on the client cert.

---

<div class="post-metadata">

**Author:** ![anoopkv](https://avatars.discourse-cdn.com/v4/letter/a/49beb7/32.png) [@anoopkv](https://discuss.elastic.co/u/anoopkv)\
**Post date:** [March 8, 2021, 9:47pm UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/13 "2021-03-08T21:47:15Z")

</div>

Hi @TimV

> [@TimV](#):
>
> That means it cannot perform user authentication based on the client cert

Here I am setting  
[ingress.kubernetes.io/ssl-passthrough](http://ingress.kubernetes.io/ssl-passthrough)

If the ingress allows ssl-passthrough I should be able to do pki auth? Or its not simply not possible with nginx that I should look for alternative load balancer

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 9, 2021, 1:18am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/14 "2021-03-09T01:18:45Z")

</div>

You are asking Kubernetes Networking questions for which I am not an expert.

k8s ssl-passthrough is designed to bypass NGINX completely by using a TCP level proxy instead which will skip nginx when it detects an SNI SSL connection to a recognised backend.

Since your Curl connection shows that your traffic is being directed to NGINX, it is clear than ssl-passthrough is not working, but I cannot tell you why (this isn't a k8s forum).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 6, 2021, 1:19am UTC](https://discuss.elastic.co/t/elasticsearch-ssl-tls-pki-realm-setup/265790/15 "2021-04-06T01:19:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.