Elasticsearch public access failure with self-signed certificate. xpack.security is it really needed

[O_K]

Hi Everyone,

Based on my previous post Elasticsearch with xpack.security.enabled throws Cluster is not yet ready

I configured xpack.security and selfsigned certificate for elasticsearch.

I wonder whether xpack.security is really needed as previously I was getting error after elasticsearch kubernetes pods restart:

ERROR: [1] bootstrap checks failed
[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]

SO, currently xpack security is enabled and I also configured certificates, but when I try to access elastic cluster from outside I'm getting below error.

Everything works fine when I access elasticsearch inside the cluster

curl -vk -u elastic:pass -GET "https://elasticsearch-master.logging.svc.cluster.local:9200/_cat/indices" --cacert ca.pem --cert elasticsearch-master.crt --key elasticsearch-master.key

But access outside of cluster throws me error:

{"type": "server", "timestamp": "2020-01-16T21:07:29,024+0000", "level": "WARN", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "s0W6l0pwRxaRRg0dxo2XXX", "node.id": "vExaDAzAT3yclCSAL47XXX",  "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/172.30.182.111:38686}" , 
"stacktrace": ["io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 47455420",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]",
"at java.lang.Thread.run(Thread.java:835) [?:?]",
"Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f66617669636f6e2e69636f204850d0a",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1206) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"... 16 more"] }

Please let me know how to deal with this issue. Should I fix certificates access or probably there is an option to disable xpack security? But will elasticsearch be stable after restarts (see error at the top)

Thank you,
OK

[nandersen674 (niels andersen)]

You need a pem certificate with the complete certificate chain in a textual format on the client side. If elastic is unable to talk to its sources, then you may need to do this at several levels.

[O_K]

Is it about to decide where tls termination should occur? I use kubernetes ingress and understand that https traffic terminates on ingress level (based on error) so just using pem on client side didn't help. There are options to passthrough tls on ingress level. Any ideas about this?

https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/