Elasticsearch SSL/TLS PKI realm setup

anoopkv · March 1, 2021, 11:58am

Hello,
I am trying to setup PKI realm on my onpremise ELK stack on a k8s cluster and I am using helm as the package manager
Elasticsearch version: 7.11.1

I have created an organization signed server certificate, using which I have enabled xpack security at transport and http layer. The certificate is password protected, so keystore values are defined as well for the following

xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
xpack.security.http.ssl.keystore.secure_password
xpack.security.http.ssl.truststore.secure_password

Here is the helm values.yaml file

  elasticsearch.yml: |

   xpack.security.enabled: true

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.verification_mode: certificate

   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.pem

   xpack.security.http.ssl.enabled: true

   xpack.security.http.ssl.verification_mode: certificate

   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/certs/ca.pem

   xpack.security.http.ssl.client_authentication: optional

   xpack.security.authc.realms.native.local.order: 0

   xpack.security.authc.realms.active_directory.my_ad.order: 1

   xpack.security.authc.realms.active_directory.my_ad.domain_name: myad

   xpack.security.authc.realms.active_directory.my_ad.url: ldap://myldap:389

   xpack.security.authc.realms.pki.pki1.order: 2

   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca.pem']

and created a role mapping in kibana

After that I created another certificate to be used by the client, signed by organization (same who signed the server certificates)

The ELK stack itself is working fine and i am able to login using as AD user, built-in user. However when i am trying to use the client certificate, it fails to authenticate. Here i am calling the authentication api from postman

error details

                "reason": "missing authentication credentials for REST request [/_xpack/security/_authenticate?pretty]",

                "header": {

                    "WWW-Authenticate": [

                        "Basic realm=\"security\" charset=\"UTF-8\"",

                        "Bearer realm=\"security\"",

                        "ApiKey"

                    ]

                }

here is a curl output

Really appreciate any pointers.

ikakavas · March 1, 2021, 1:39pm

Your curl reproduction contains --cacert twice, the first one (pointing to client.cer) needs to be --cert instead

anoopkv · March 1, 2021, 2:58pm

Thanks @ikakavas

Couldnt get curl to work. While i am still at it - Do you see any issues with the setup and/or the error in postman ?

anoopkv · March 1, 2021, 3:47pm

Not sure if i am doing it right, but here it is. getting the same error like i saw in postman
WWW-Authenticate shy is it not showing pki realm. do you see any issues in my setup ?

TimV · March 2, 2021, 3:06am

The most likely explanation is that the server cannot establish is not a trust path between client.crt and ca.pem

You have a couple of options to try and debug that:

Use external tools such as openssl verify to check that the client certificate verifies against the configured certificate authority
Switch xpack.security.http.ssl.client_authentication to required and check the ES log for errors. Debugging optional client auth is hard because there is no error when it fails.

Also, your curl commands are all passing --cacert as well as -k, which is telling curl to verify the server against ca.cer, but also don't verify it.
It would be better for you to drop the -k and get it working with --cacert so that you know you are verifying the correct behaviour.

anoopkv · March 2, 2021, 12:38pm

After setting client_authentication to required, I get error within elasticsearch node

"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
"at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]",
"at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]",
"at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"... 16 more"] }

Was able to verify the client certificate

here is the changed yaml (added root, intermediate ca to certificate authorities list)

elasticsearch.yml: |

   xpack.security.enabled: true

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.verification_mode: certificate

   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.transport.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']

   xpack.security.http.ssl.enabled: true

   xpack.security.http.ssl.verification_mode: certificate

   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']

   xpack.security.http.ssl.client_authentication: required

   xpack.security.authc.realms.native.local.order: 0

   xpack.security.authc.realms.active_directory.my_ad.order: 1

   xpack.security.authc.realms.active_directory.my_ad.domain_name: adep01.nordic.rsa-ins.com

   xpack.security.authc.realms.active_directory.my_ad.url: ldap://ADEP01.nordic.rsa-ins.com:389

   xpack.security.authc.realms.pki.pki1.order: 2

   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/root-ca.pem', '/usr/share/elasticsearch/config/certs/intermediate-ca.pem']

TimV · March 3, 2021, 1:50am

I think this is your issue.

You are listing each element in the signing chain as a separate item in the certificate_authorities array, but that's not how it works.

In fact, each file that is listed in certificate_authorities is a supposed to be a chain in itself. The array structure allows you to trust multiple CA chains, not to specify each certificate individually.

Instead you need to merge those certificates into a single PEM cert chain file and list that.

cat /usr/share/elasticsearch/config/certs/intermediate-ca.pem \
    /usr/share/elasticsearch/config/certs/root-ca.pem \
  > /usr/share/elasticsearch/config/certs/ca-chain.pem

xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

anoopkv · March 3, 2021, 6:10am

If you see my initial setup, I had only kept one certificate - root ca.
Anyways I have tried your suggestion

type D:\Certificates\elk\intermediate-ca.pem D:\Certificates\elk\root-ca.pem > D:\Certificates\elk\ca-chain.pem

elasticsearch.yml: |

   xpack.security.enabled: true

   xpack.security.transport.ssl.enabled: true

   xpack.security.transport.ssl.verification_mode: certificate

   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.transport.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

   xpack.security.http.ssl.enabled: true

   xpack.security.http.ssl.verification_mode: certificate

   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate2.p12

   xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

   xpack.security.http.ssl.client_authentication: required

   xpack.security.authc.realms.native.local.order: 0

   xpack.security.authc.realms.active_directory.my_ad.order: 1

   xpack.security.authc.realms.active_directory.my_ad.domain_name: adep01.nordic.rsa-ins.com

   xpack.security.authc.realms.active_directory.my_ad.url: ldap://ADEP01.nordic.rsa-ins.com:389

   xpack.security.authc.realms.pki.pki1.order: 2

   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

And i get the exact same error as last time ( i am not pasting the full error again, as it is same as earlier)
"Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain"

Not sure if it is of any info, when I try to run openssl verify, I get error

anoopkv · March 6, 2021, 2:06pm

Hi @TimV

Just for test I am trying to use the same server certificate.
I am able to verify it using openssl

Using that certificate and key, I try to do a curl and get error for "missing authentication credentials"

my current config values.

  elasticsearch.yml: |
   xpack.security.enabled: true
   xpack.security.transport.ssl.enabled: true
   xpack.security.transport.ssl.verification_mode: certificate
   xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate.p12
   xpack.security.transport.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']
   xpack.security.http.ssl.enabled: true
   xpack.security.http.ssl.verification_mode: certificate
   xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elk_certificate.p12
   xpack.security.http.ssl.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']
   xpack.security.http.ssl.client_authentication: optional
   xpack.security.authc.realms.native.local.order: 0
   xpack.security.authc.realms.active_directory.my_ad.order: 1
   xpack.security.authc.realms.active_directory.my_ad.domain_name: adep01.nordic.rsa-ins.com
   xpack.security.authc.realms.active_directory.my_ad.url: ldap://myldap:389
   xpack.security.authc.realms.pki.pki1.order: 2
   xpack.security.authc.realms.pki.pki1.certificate_authorities: ['/usr/share/elasticsearch/config/certs/ca-chain.pem']

Why does my cluster fail, when i make client_authentication "required". Only elastic was running in my cluster
Is my realm setup correct?

I am stuck with issue for more than two weeks now . Any idea what could be wrong ?

TimV · March 7, 2021, 2:22am

Your curl output (which I can't quote here because you pasted a screenshot - please don't do that, you should copy the text directly) says it is using HTTP/2

Elasticsearch doesn't support HTTP/2, so if the curl output is to be believed, then whatever you are connecting too isn't an Elasticsearch node. Do you have a proxy or load balancer in place?

If you have a paid license then you should contact support, they can give you far more effective support than we can.

anoopkv · March 7, 2021, 3:05pm

Hi @TimV

Thanks a lot for that pointer, you are spot on. My entire ELK stack is setup on a k8s cluster, and I am using an nginx controller as the ingress

When I enable port-forwarding and do a curl. it seems to work fine
So afterall the PKI setup, the client and server certificates all are correct
C:\Program Files\Git\usr\bin>curl.exe -v -key C:\cert\elk_key.pem -cert C:\cert\elk_certificate.crt https://127.0.0.1:9200/_xpack/security/_authenticate
* Rebuilt URL to: C:\cert\elk_key.pem/
* Port number ended with ''
* Closing connection -1
curl: (3) Port number ended with ''
* Rebuilt URL to: C:\cert\elk_certificate.crt/
* Port number ended with ''
* Closing connection -1
curl: (3) Port number ended with ''
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: using IP address, SNI is not supported by OS.
* schannel: sending initial handshake data: sending 147 bytes...
* schannel: sent initial handshake data: sent 147 bytes
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5120 length 5120
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 6144 length 6144
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 62
* schannel: encrypted data buffer: offset 6206 length 7168
* schannel: sending next handshake data: sending 4649 bytes...
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 7168
* schannel: encrypted data buffer: offset 7168 length 7168
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 8192 length 8192
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 9216 length 9216
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 2/3)
* schannel: encrypted data got 945
* schannel: encrypted data buffer: offset 10161 length 10240
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with 127.0.0.1 port 9200 (step 3/3)
* schannel: stored credential handle in session cache
> GET /_xpack/security/_authenticate HTTP/1.1
> Host: 127.0.0.1:9200
> User-Agent: curl/7.55.1
> Accept: /
> Referer: y
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 523
* schannel: encrypted data buffer: offset 523 length 103424
* schannel: decrypted data length: 494
* schannel: decrypted data added: 494
* schannel: decrypted data cached: offset 494 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 494 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 494
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< Warning: 299 Elasticsearch-7.11.1-ff17057114c2199c9c1bbecc727003a907c0db7a "[GET /_xpack/security/_authenticate] is deprecated! Use [GET /_security/_authenticate] instead."
< content-type: application/json; charset=UTF-8
< content-length: 233
<
{"username":"uid1","roles":,"full_name":null,"email":null,"metadata":{"pki_dn":"CN=uid1"},"enabled":true,"authentication_realm":{"name":"pki1","type":"pki"},"lookup_realm":{"name":"pki1","type":"pki"},"authentication_type":"realm"}* Connection #0 to host 127.0.0.1 left intact

When i curl it directly it is not working

C:\Program Files\Git\usr\bin>curl.exe -v -key C:\cert\elk_key.pem -cert  C:\cert\elk_certificate.crt https://elasticsearch.mydomain.com/_xpack/security/_authenticate
* Rebuilt URL to: C:\cert\elk_key.pem/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
* Rebuilt URL to: C:\cert\elk_certificate.crt/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
*   Trying 10.204.169.6...
*   Trying 10.204.169.6...
* TCP_NODELAY set
* Connected to elasticsearch.mydomain.com (10.204.169.6) port 443 (#0)
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 1/3)
* schannel: disabled server certificate revocation checks
* schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.
* schannel: sending initial handshake data: sending 195 bytes...
* schannel: sent initial handshake data: sent 195 bytes
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 2/3)
* schannel: encrypted data got 2752
* schannel: encrypted data buffer: offset 2752 length 4096
* schannel: sending next handshake data: sending 4649 bytes...
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with elasticsearch.mydomain.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /_xpack/security/_authenticate HTTP/1.1
> Host: elasticsearch.mydomain.com
> User-Agent: curl/7.55.1
> Accept: */*
> Referer: y
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 895
* schannel: encrypted data buffer: offset 895 length 103424
* schannel: decrypted data length: 866
* schannel: decrypted data added: 866
* schannel: decrypted data cached: offset 866 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 866 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 866
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401 Unauthorized
< Date: Sun, 07 Mar 2021 09:29:26 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 517
< Connection: keep-alive
< WWW-Authenticate: Basic realm="security" charset="UTF-8"
< WWW-Authenticate: Bearer realm="security"
< WWW-Authenticate: ApiKey
< Strict-Transport-Security: max-age=15724800; includeSubDomains
<
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}* Connection #0 to host elasticsearch.mydomain.com left intact

Tried the setup suggested, but does not seem to make any difference.
nginx controller setup May be its not the right forum, but Can you please take a look at my setup

ingress:
  enabled: true
  annotations:
    ingress.kubernetes.io/ssl-passthrough: "true"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-secret: "default/ca-secret2"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
  path: /
  hosts:
    - elasticsearch.mydomain.com
  tls:
    - secretName: elastic-tls
      hosts:
        - elasticsearch.mydomain.com
  rules:
  - host: elasticsearch.mydomain.com
    http:
      paths:
      - path: /
        backend:
          serviceName: elasticsearch-master
          servicePort: 9200

I am in catch22 situation, i need to complete a POC and evaluate it before my org can buy the license

TimV · March 7, 2021, 11:34pm

You cannot perform PKI authentication through an HTTP layer proxy. In your case nginx is terminating TLS, so Elasticsearch never sees the client certificate. That means it cannot perform user authentication based on the client cert.

anoopkv · March 8, 2021, 9:47pm

Hi @TimV

Here I am setting
ingress.kubernetes.io/ssl-passthrough

If the ingress allows ssl-passthrough I should be able to do pki auth? Or its not simply not possible with nginx that I should look for alternative load balancer

TimV · March 9, 2021, 1:18am

You are asking Kubernetes Networking questions for which I am not an expert.

k8s ssl-passthrough is designed to bypass NGINX completely by using a TCP level proxy instead which will skip nginx when it detects an SNI SSL connection to a recognised backend.

Since your Curl connection shows that your traffic is being directed to NGINX, it is clear than ssl-passthrough is not working, but I cannot tell you why (this isn't a k8s forum).

system · April 6, 2021, 1:19am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ELK with Xpack Elasticsearch	8	7035	March 6, 2020
ElasticSearch XPack - PKI Realm authentication defaults to native authentication Elasticsearch elastic-stack-security	4	2356	May 25, 2020
ELK Stack + TLS + Headache Kibana elastic-stack-security	15	2123	September 17, 2021
I'm struggling set up the minimal Security and the configure the TLS Elasticsearch elastic-stack-security	6	626	January 20, 2023
Elasticsearch security PKI Elasticsearch elastic-stack-security	13	2196	September 19, 2019

Elasticsearch SSL/TLS PKI realm setup

Related topics