Problem setting up cluster with NAT address

pmartins · June 17, 2014, 12:17pm

Hi,

I'm having some problems setting up a 1.2.1 ES cluster. I have two nodes, each one in a different data center/network.

One of the nodes is behind a NAT address, so I set network.publish_host to de NAT address.

Both nodes connect to each other without problems. The issue is when the node behind the NAT address tries to connect to himself. In my network, he doesn't know his NAT address and can't solve it. So I get the exception:

[2014-06-17 12:58:19,681][WARN ][cluster.service ] [vm-motisqaapp02] failed to reconnect to node [vm-motisqaapp02][4oSfsIaBTSyQWdnxiTt7Cw][vm-motisqaapp02.***][inet[/10.10.1.135:9300]]{master=true}
org.elasticsearch.transport.ConnectTransportException: [vm-motisqaapp02][inet[/10.10.1.135:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:656)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:624)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:146)
at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:518)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException: connection timed out: /10.10.1.135:9300
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more

vm-motisqaapp02 NAT address is 10.10.1.135, but locally it can't solve this address. Is there any way that I can setup other IP to comunicate locally?

Georgi_Ivanov · June 17, 2014, 1:59pm

Doesn't sound like elasticsearch issue ...

I would look to my FW rules

On Tuesday, June 17, 2014 2:17:20 PM UTC+2, pmartins wrote:

Hi,

I'm having some problems setting up a 1.2.1 ES cluster. I have two nodes,
each one in a different data center/network.

One of the nodes is behind a NAT address, so I set network.publish_host to
de NAT address.

Both nodes connect to each other without problems. The issue is when the
node behind the NAT address tries to connect to himself. In my network, he
doesn't know his NAT address and can't solve it. So I get the exception:

[2014-06-17 12:58:19,681][WARN ][cluster.service ]
[vm-motisqaapp02] failed to reconnect to node
[vm-motisqaapp02][4oSfsIaBTSyQWdnxiTt7Cw][vm-motisqaapp02.***][inet[/10.10.1.135:9300]]{master=true}

org.elasticsearch.transport.ConnectTransportException:
[vm-motisqaapp02][inet[/10.10.1.135:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
    at 
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:656)
    at 
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:624)
    at 
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:146)
    at 
org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:518)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.common.netty.channel.ConnectTimeoutException:
connection timed out: /10.10.1.135:9300
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    at 
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at 
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at 
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at 
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at 
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    ... 3 more 
vm-motisqaapp02 NAT address is 10.10.1.135, but locally it can't solve
this
address. Is there any way that I can setup other IP to comunicate locally?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Problem-setting-up-cluster-with-NAT-address-tp4057849.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/24e11c67-8133-4893-b665-09f31735f269%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

pmartins · June 17, 2014, 2:12pm

Hi,

Thanks for the reply.

The firewall on the node is off, and he can't comunicate with himself. The problem is:

vm-motisqaapp02 has the local address 172.16.3.81 with the NAT 10.10.1.135. But, with the current data center definitions, it can't solve the 10.10.1.135 doesn't recognizing itself.

Can I configure different adresses for network.publish host ? One for comunicating with outside nodes and another with itself?

warkolm · June 17, 2014, 11:26pm

You can only define one address for ES to use.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 18 June 2014 00:12, pmartins pedro.martins@spms.min-saude.pt wrote:

Hi,

Thanks for the reply.

The firewall on the node is off, and he can't comunicate with himself. The
problem is:

vm-motisqaapp02 has the local address 172.16.3.81 with the NAT 10.10.1.135.
But, with the current data center definitions, it can't solve the
10.10.1.135 doesn't recognizing itself.

Can I configure different adresses for network.publish host ? One for
comunicating with outside nodes and another with itself?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Problem-setting-up-cluster-with-NAT-address-tp4057849p4057867.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1403014356232-4057867.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Yt__EfwsqM%2B4k8goMCdwCnwZ21RQjF4C%2BuAzNZrYnnHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

pmartins · June 18, 2014, 9:25am

So, just to be clear:

If we have a ES cluster in two different networks and a node doesn't "know" his foreign address (NAT), there's no way to configure the transport layer on cluster to comunicate without problems?

Thank you for your answers.

warkolm · June 18, 2014, 9:27am

No, that's outside the scope of ES to understand.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 18 June 2014 19:25, pmartins pedro.martins@spms.min-saude.pt wrote:

So, just to be clear:

If we have a ES cluster in two different networks and a node doesn't "know"
his foreign address (NAT), there's no way to configure the transport layer
on cluster to comunicate without problems?

Thank you for your answers.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Problem-setting-up-cluster-with-NAT-address-tp4057849p4057921.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1403083531235-4057921.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bfj-XW7BW8q3%3DrM2aWQt%2B5nA0GZDSoGhe9-Ly6tbdYCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

pmartins · June 18, 2014, 12:05pm

OK, that means that ES clusters don't work with nodes in different networks, when at least one of them has a NAT address to comunicate with the other, right?

If so, I need to change the architecture of my solution.

Thank you for all the help