I'm having trouble configuring TLS for es-hadoop.
Running 7.2 Elasticsearch in a 3 node cluster (1 data, 1 master, 1 coordinating which hosts kibana).
Running Spark on an EMR cluster in the same vpn. Before enabling xpack security, was able to connect to ES from Spark using both es-hadoop and Rest Client.
Configured TLS with CA and single signing cert, generated by certutil, used in all es and spark nodes.
The nodes all ES nodes communicate with each other correctly.
But in spark, the es-hadoop connector can no longer connect to the ES cluster. (The rest client running in the spark job can connect using basic auth). For experimentation, we are running this job from a zeppelin notebook spark interpreter.
I am using the same certificate in the spark cluster nodes which I am using in the ES cluster nodes.
spark es config properties are:
es.net.http.auth.pass : xxxxxx
es.net.http.auth.user : xxxxxx
es.net.ssl : true
es.net.ssl.keystore.location : file:///home/hadoop/ria/certs/test-es-node-cert.p12
es.net.ssl.trustore.location : file:///home/hadoop/ria/certs/test-es-node-cert.p12
es.nodes : 10.0.1.179
es.port : 9300
Rest client works fine, connector fails. In the spark logs on failure, message begins:
%text org.elasticsearch.hadoop.EsHadoopIllegalArgumentExc eption: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:340) at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:220) at
in es logs, stack trace is
exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.0.1.81:51709}], closing connection io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: internal_error at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final] at java.lang.Thread.run(Thread.java:835) [?:?] Caused by: javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final] ... 16 more [2019-08-28T14:12:16,592][WARN ][o.e.t.TcpTransport ] [test-es-data-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.0.1.81:39683}], closing connection io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: internal_error at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final] at 4
elided....
Are there some other additional or different configurations required? Do I need a different certificate to act as the es-hadoop keystore? Any ideas would be helpful.