Both elasticsearch hosts are running 7.3.2. The scenario is that I am using the transport profile settings to connect the nodes using a certificate, created by following the setting up elastic security blog post. I can verify that the transport profile on the master is working correctly as an internal es host is connected properly to the default profile, while the so called external es host packet traces show the connection being established, using the secondary profile.
The external es host will not complete cluster binding with the error log on the es master host showing the SSLHandshakeException: No available authentication scheme. This external es host is using the same cluster certificate as other hosts that already successfully joined the cluster.
`[2019-09-20T16:13:52,846][WARN ][o.e.t.TcpTransport ] [elastic1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9301, remoteAddress=/xxx.xxx.xxx.xxx:34636}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No available authentication scheme
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: No available authentication scheme
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateProducer.onProduceCertificate(CertificateMessage.java:944) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateProducer.produce(CertificateMessage.java:933) ~[?:?]
at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436) ~[?:?]
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.goServerHello(ClientHello.java:1225) ~[?:?]
at sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(ClientHello.java:1161) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:852) ~[?:?]
at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:813) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:689) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
... 16 more `
# ---------------------------------- Network -----------------------------------#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#for local nodes
transport.profiles.default.port: 9300
transport.profiles.default.bind_host: 0.0.0.0
transport.profiles.default.publish_host: 172.18.1.48
# for ext. nodes
transport.profiles.dc.port: 9301
transport.profiles.dc.bind_host: 0.0.0.0
transport.profiles.dc.publish_host: xx.xx.xx.xx
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
## Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
discovery.seed_hosts: ["elastic1", "elastic2", "elastic3", "elasticdc1"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["elastic1", "elastic2"]
#
# For more information, consult the discovery and cluster formation module documentation.
## ---------------------------------- Gateway -----------------------------------
## Block initial recovery after a full cluster restart until N nodes are started:
# gateway.recover_after_nodes: 3
## For more information, consult the gateway module documentation.#
# ---------------------------------- Various -----------------------------------
## Require explicit names when deleting indices:
##action.destructive_requires_name: true
# ENABLES SEC. - autogens an elastic user password
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.license.self_generated.type: basic
Int Host:
# ---------------------------------- Network -----------------------------------## Set the bind address to a specific IP (IPv4 or IPv6):#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["elastic1", "elastic2", "elastic3"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["elastic1", "elastic2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
# gateway.recover_after_nodes: 3
## For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
# ENABLES SEC. - autogens an elastic user password
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.license.self_generated.type: basic
Ext Host:
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
## bind for main cluster
transport.profiles.default.port: 9300
transport.profiles.default.bind_host: 0.0.0.0
transport.profiles.default.publish_host: xx.xx.xx.xx
#transport for local
transport.profiles.dc.port: 9301
transport.profiles.dc.bind_host: 0.0.0.0
transport.profiles.dc.publish_host: xx.xx.xx.xx
# For more information, consult the network module documentation.## --------------------------------- Discovery ---------------------------------- ## Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["elastic1:9301"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["elastic1:9301"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
# ENABLES SEC. - autogens an elastic user password
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.license.self_generated.type: basic
each have a copy of the correct cert. verified through /_ssl/certificates
Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.
Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.
It would be great if you could update your post to solve this.
Can you explain why you want to use transport profiles, especially since both use the same settings? The TCP transport profiles main use case was to separate node-node and node-client traffic and were deprecated in 7.3 as they will be removed in future versions.
Testing an external ingest node to relay data to the main cluster. The IP address is different as it passes through a FW. During troubleshooting, I noticed that when ext. host would initially make contact, it would receive the internal address of the master node. When adding the transport profile, it was able to communicate with the master over the specified FW IP. If there is a better method for this approach please let me know.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.