Connection Refused fun continues - is there anyone I can pay for support?

[newsgroup (newsgroup@syslog.com)]

The random connection issues continue to plague my production
systems. Now that I've brought up a second remote server, I'm getting
the following errors:

[2012-04-18 10:13:50,398][WARN ][cluster.service ] [app1-uk-
example-com] failed to reconnect to node [app2-uk-example-com]
[HRCQGb4lTIm-hjzENY3mTQ][inet[/172.28.5.3:9300]]{zone=uk}
org.elasticsearch.transport.ConnectTransportException: [app2-uk-
example-com][inet[/172.28.5.3:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:
560)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
503)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
482)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:
128)
at org.elasticsearch.cluster.service.InternalClusterService
$ReconnectToNodes.run(InternalClusterService.java:377)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:400)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:362)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:284)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
102)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:42)

So the two UK servers won't talk to each other, and the status info on
app1 show green but no shards, and the status info on app2 shows
proper sharding and a green cluster.

I'm happy to pay someone to help me debug this problem - does anyone
offer paid ES support?

[otisg (Otis Gospodnetić)]

Hello,

Sematext offers Elasticsearch consulting services and support:

• Solr, Elasticsearch & OpenSearch Production Support - Sematext
• Solr, Elasticsearch & OpenSearch Production Support - Sematext

Otis

On Wednesday, April 18, 2012 10:20:20 AM UTC-4, news...@syslog.com wrote:

The random connection issues continue to plague my production
systems. Now that I've brought up a second remote server, I'm getting
the following errors:

[2012-04-18 10:13:50,398][WARN ][cluster.service ] [app1-uk-
example-com] failed to reconnect to node [app2-uk-example-com]
[HRCQGb4lTIm-hjzENY3mTQ][inet[/172.28.5.3:9300]]{zone=uk}
org.elasticsearch.transport.ConnectTransportException: [app2-uk-
example-com][inet[/172.28.5.3:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:

560. at
     org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:

561. at
     org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:

562. at
     org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:

563. at org.elasticsearch.cluster.service.InternalClusterService
     $ReconnectToNodes.run(InternalClusterService.java:377)
     at
     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

564. at java.util.concurrent.ThreadPoolExecutor
     $Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:679)
     Caused by: java.net.ConnectException: Connection refused
     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
     at
     sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
     at
     org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink

$Boss.connect(NioClientSocketPipelineSink.java:400)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink

$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:362)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink

$Boss.run(NioClientSocketPipelineSink.java:284)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:

102. at
     org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
     $1.run(DeadLockProofWorker.java:42)

So the two UK servers won't talk to each other, and the status info on
app1 show green but no shards, and the status info on app2 shows
proper sharding and a green cluster.

I'm happy to pay someone to help me debug this problem - does anyone
offer paid ES support?

[kimchy (Shay Banon)]

Do you have any firewall or something that can block the two nodes from
talking to one antoher?

On Wed, Apr 18, 2012 at 5:20 PM, newsgroup@syslog.com
newsgroup@syslog.comwrote:

The random connection issues continue to plague my production
systems. Now that I've brought up a second remote server, I'm getting
the following errors:

[2012-04-18 10:13:50,398][WARN ][cluster.service ] [app1-uk-
example-com] failed to reconnect to node [app2-uk-example-com]
[HRCQGb4lTIm-hjzENY3mTQ][inet[/172.28.5.3:9300]]{zone=uk}
org.elasticsearch.transport.ConnectTransportException: [app2-uk-
example-com][inet[/172.28.5.3:9300]] connect_timeout[30s]
at

org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:
560)
at

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
503)
at

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
482)
at

org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:
128)
at org.elasticsearch.cluster.service.InternalClusterService
$ReconnectToNodes.run(InternalClusterService.java:377)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.connect(NioClientSocketPipelineSink.java:400)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:362)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.run(NioClientSocketPipelineSink.java:284)
at

org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:
102)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker
$1.run(DeadLockProofWorker.java:42)

So the two UK servers won't talk to each other, and the status info on
app1 show green but no shards, and the status info on app2 shows
proper sharding and a green cluster.

I'm happy to pay someone to help me debug this problem - does anyone
offer paid ES support?

[newsgroup (newsgroup@syslog.com)]

On Apr 19, 10:55 am, Shay Banon kim...@gmail.com wrote:

Do you have any firewall or something that can block the two nodes from
talking to one antoher?

No, there was no firewall. The UK servers had multiple aliased IPs
and the ES server was taking one of the aliases because I did not have
a specific IP configured. This works on boxed with only one IP but
not multiple. Once I put the non-alias IP in the config, they could
talk to each other.

The aliased IP work fine otherwise for other servers.