TLS On Elasticsearch REST API For Use With External Clients

Jstall · November 27, 2019, 1:26pm

Hello all,

My use case is I would like to secure the Elastic Search REST API with tls so that I can post to it securely from an external client.
I am using a ElasticSearch 7.4 cluster that I deployed via the official Marketplace app on Azure and I have a single data node as I wanted
to keep this as simple as I could for the prototype/proof of concept I am working on.

The client is a .NET Core application and I'm using the serilog elasticsearch sink to connect.

I am using certificates that I generated with elasticsearch-certutil. This is what I did:

# create a ca called elastic-stack-ca.p12
elasticsearch-certutil ca
# create certificate archieve called elastic-certificates.p12 using the ca we just created
elasticsearch-certutil cert --ca elastic-stack-ca.p12

I moved these to the directory /etc/elasticsearch/certs.

In my elasticsearch.yml:

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.type: PKCS12
xpack.security.http.ssl.truststore.type: PKCS12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: required

I then generated client certs using the same ca the server certs used:
elasticsearch-certutil cert --ca /etc/elasticsearch/certs/elastic-stack-ca.p12 --out client.p12

I can, using Postman or my .NET Core app successfully send a request to my elasticsearch instance using the client.p12
file. What I'm wondering though, is it possible to allow authentication using just the public cert or even the ca cert
that is packaged in client.p12. If I extract them via openssl:

openssl pkcs12 -in client.p12 -cacerts -nokeys -chain > client-ca.cer
openssl pkcs12 -in client.p12 -clcerts -nokeys  > client.cer

and try to just use the public certificate in my post to ES, again either Postman or .NET client, I get no response from the server and see the following in my elasticsearch logs:

[2019-11-27T12:43:45,528][WARN ][o.e.h.AbstractHttpServerTransport] [data-0] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.17.5.6:9200, remoteAddress=/165.225.38.246:6398}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[netty-codec-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) [netty-transport-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar:4.1.38.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar:4.1.38.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:258) ~[?:?]
        at 
    ... 16 more

This error puzzles me because it the message "Empty server certificate chain" suggests the problem is on my elasticsearch server.

My question is:
Does a client have to have an entire .p12 file that includes ca-cert/privatekey/public-cert in order to authenticate with tls on ES? Is it possible to securely communicate with, for instance, just the ca-cert and/or public cert? Am I misconfiguring something on my es instance?

One of the reasons I am asking this is because I am finding the client-side validation of the cert returned by elasticsearch to be a bit involved. I am essentially pulling the ca out of the client.p12 file and comparing it to the first element in the chain returned by es, then attempting to rebuild
the chain using the certificate elasticsearch sends back and checking for errors. It would be easier if I didn't have to dig for the ca cert on the client side. Also I would like to make this work with the minimum amount of information on the client side so I would like to ensure I am doing things efficiently here.

Sorry for the wall of text. Any advice anyone could give in this regard would be appreciated. Thanks much!

ikakavas · November 28, 2019, 6:14am

Hi there @Jstall

My use case is I would like to secure the Elastic Search REST API with tls so that I can post to it securely from an external client.

TLS offers potentially several security properties.

It can be used to authenticate the server to the client ( the server uses a key and a corresponding certificate from a certificate authority that the client trusts saying that this is X , so it must be X )
It can be used to authenticate the client to the server ( the client uses a key and a corresponding certificate from a certificate authority that the server trusts. The certificate identifies the client as Y so, it must be Y )
it also encrypts the communications between the client and the server offering integrity protection, confidentiality etc.

Which of the above are included in your "secure with tls" ?

Have you decided how do you want to authenticate your client to the elasticsearch REST API ? Your initial configuration (xpack.security.http.ssl.client_authentication: required) suggests that you want to do client TLS authentication ( that is have your client authenticate with a certificate and key ) but your questions later allude to the fact that you may not want to do it after all. You don't have to do client TLS authentication, you could use credentials ( ie username and password )

If you want to do TLS Client authentication then the client needs its own private key and a certificate. These can be in a keystore ( like PKCS#12 that you have here ) or you can export them from the keystore as PEM file. The client certificate is not enough on its own to authenticate the client. Also the CA certificate is not something that can be used to authenticate the client.

Now if you don't want to do client TLS authentication, and you only need to authenticate the server during the TLS handshake, then all you need in the client is the CA certificate that has signed the servers certificate so that you can verify and trust that the server is who it says it is. You would also need to change xpack.security.http.ssl.client_authentication: required to xpack.security.http.ssl.client_authentication: optional or just remove it.

Hope this helps

system · December 26, 2019, 6:14am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.