Unicast Cluster / Host can't see its own publish_host

Elastic Stack Elasticsearch
1

In my VM environment, a VM can't actually see its public IP address. I
have the following setup:

network.publish_host: 10.255.207.123
discovery.zen.ping.unicast.hosts: 10.255.207.123,10.255.207.124,10.255.
207.125

My VM can see 124 and 125 just fine, but due to issues completely unrelated
to ES, it cannot see its own public IP. As a result, the logs on each
machine fill up with these exceptions:

org.elasticsearch.transport.ConnectTransportException: [elk2][inet[/10.255.207.123:9300]]
connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at
org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:521)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:
connection timed out: /10.255.207.123:9300
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
processConnectTimeout(NioClientBoss.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process
(NioClientBoss.java:83)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.
run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(
DeadLockProofWorker.java:42)

I believe this is pretty innocuous as I've been running this way for 6
months, but I've always been curious as to way my node is even trying to
connect to itself like this? As it goes through the list of
'discovery.zen.ping.unicast.hosts', shouldn't it ignore itself? Maybe ES
doesn't use 'network.publish_host' to determine if the IP belongs to the
current machine?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fa2e66cf-1f96-4c9c-8b69-bd146f868ec7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

It sounds like because that isn't a local interface that ES is bound to it
tries to access it. Are you using NAT on a higher layer?

On 6 January 2015 at 01:59, Matt Hughes hughes.matt@gmail.com wrote:

In my VM environment, a VM can't actually see its public IP address. I
have the following setup:

network.publish_host: 10.255.207.123
discovery.zen.ping.unicast.hosts: 10.255.207.123,10.255.207.124,10.255.
207.125

My VM can see 124 and 125 just fine, but due to issues completely
unrelated to ES, it cannot see its own public IP. As a result, the logs on
each machine fill up with these exceptions:

org.elasticsearch.transport.ConnectTransportException: [elk2][inet[/
10.255.207.123:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at
org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:521)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:
connection timed out: /10.255.207.123:9300
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
processConnectTimeout(NioClientBoss.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
process(NioClientBoss.java:83)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector
.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.
run(DeadLockProofWorker.java:42)

I believe this is pretty innocuous as I've been running this way for 6
months, but I've always been curious as to way my node is even trying to
connect to itself like this? As it goes through the list of
'discovery.zen.ping.unicast.hosts', shouldn't it ignore itself? Maybe ES
doesn't use 'network.publish_host' to determine if the IP belongs to the
current machine?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fa2e66cf-1f96-4c9c-8b69-bd146f868ec7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fa2e66cf-1f96-4c9c-8b69-bd146f868ec7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-4Zk90wSrM8Z6oqHR1LGNdLLSB4vcYVqdPp%2BvLWoRMgA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Well ES is bound to 0.0.0.0. Each node gets traffic from other nodes, it
just can't communicate with its own public IP.

My guess is that on normal systems where
NetworkInterface.getNetworkInterfaces returns 10.255.207.123, Elasticsearch
would not even try to connect to itself when going over the list of
unicast.hosts. The problem -- I presume -- is that it should also use
'network.publish_host', e.g.:

val ipsOfSelf = NetworkInterface.getNetworkInterfaces.map ( _.getIp ).toSet
unicastHosts.foreach { host =>
if (!ipsOfSet.contains(host) && host != publishHostIp) {
// check connectivity to other host in the cluster
}
}

On Monday, January 5, 2015 5:42:55 PM UTC-5, Mark Walkom wrote:

It sounds like because that isn't a local interface that ES is bound to it
tries to access it. Are you using NAT on a higher layer?

On 6 January 2015 at 01:59, Matt Hughes <hughe...@gmail.com <javascript:>>
wrote:

In my VM environment, a VM can't actually see its public IP address. I
have the following setup:

network.publish_host: 10.255.207.123
discovery.zen.ping.unicast.hosts: 10.255.207.123,10.255.207.124,10.255.
207.125

My VM can see 124 and 125 just fine, but due to issues completely
unrelated to ES, it cannot see its own public IP. As a result, the logs on
each machine fill up with these exceptions:

org.elasticsearch.transport.ConnectTransportException: [elk2][inet[/
10.255.207.123:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
at
org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:521)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Caused by:
org.elasticsearch.common.netty.channel.ConnectTimeoutException: connection
timed out: /10.255.207.123:9300
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
processConnectTimeout(NioClientBoss.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.
process(NioClientBoss.java:83)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(
NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.
run(DeadLockProofWorker.java:42)

I believe this is pretty innocuous as I've been running this way for 6
months, but I've always been curious as to way my node is even trying to
connect to itself like this? As it goes through the list of
'discovery.zen.ping.unicast.hosts', shouldn't it ignore itself? Maybe ES
doesn't use 'network.publish_host' to determine if the IP belongs to the
current machine?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fa2e66cf-1f96-4c9c-8b69-bd146f868ec7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fa2e66cf-1f96-4c9c-8b69-bd146f868ec7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/af1e823d-1695-4049-80d9-d00e7e84312d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.