Connection Refused after a while, restarting helps

[johanhaest]

Hi,

I'm running the following version of elasticsearch on my ubuntu server.

curl localhost:9200
{
  "status" : 200,
  "name" : "Desmond Pitt",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.4.5",
    "lucene_version" : "4.10.4",
    ....
  }
}

But after a while (sometimes months sometimes days) I get a connection refused on elasticsearch.
When i just restart the service, it suddenly works again.
But since this runs in production I don't want the elasticsearch service to suddenly break.
It runs on localhost (port 9200), on the same server where the rest of the code resides.

Running curl localhost:9200 also returns a connection refused.

When I look into the logs I see these errors, ES stopped working on ~10:50 when i Restarted at 11:09. The [WARN ][cluster.service ] [George Washington Bridge] failed to reconnect to node seems to constantly appear but has no effect.

[2016-11-15 10:44:48,789][WARN ][cluster.service          ] [George Washington Bridge] failed to reconnect to node [George Washington Bridge][y0XjMTweT-CTRyGtLFObhA][bitnami-nginxstack-50be][inet[/100.114.244.37:9300]]
org.elasticsearch.transport.ConnectTransportException: [George Washington Bridge][inet[/100.114.244.37:9300]] connect_timeout[30s]
	at 
	...
org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	... 3 more
[2016-11-15 11:09:10,252][INFO ][node                     ] [Desmond Pitt] version[1.4.5], pid[33442], build[2aaf797/2015-04-27T08:06:06Z]
[2016-11-15 11:09:10,260][INFO ][node                     ] [Desmond Pitt] initializing ...
[2016-11-15 11:09:10,303][INFO ][plugins                  ] [Desmond Pitt] loaded [], sites []
[2016-11-15 11:09:18,785][INFO ][node                     ] [Desmond Pitt] initialized
[2016-11-15 11:09:18,786][INFO ][node                     ] [Desmond Pitt] starting ...
[2016-11-15 11:09:19,241][INFO ][transport                ] [Desmond Pitt] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/100.114.244.37:9300]}
[2016-11-15 11:09:19,338][INFO ][discovery                ] [Desmond Pitt] elasticsearch/2-uc6WDiRwSTwsSXf8jg6A
[2016-11-15 11:09:23,185][INFO ][cluster.service          ] [Desmond Pitt] new_master [Desmond Pitt][2-uc6WDiRwSTwsSXf8jg6A][bitnami-nginxstack-50be][inet[/100.114.244.37:9300]], reason: zen-disco-join (elected_as_master)
[2016-11-15 11:09:23,461][INFO ][http                     ] [Desmond Pitt] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/100.114.244.37:9200]}
[2016-11-15 11:09:23,462][INFO ][node                     ] [Desmond Pitt] started
[2016-11-15 11:09:25,327][INFO ][gateway                  ] [Desmond Pitt] recovered [6] indices into cluster_state
[2016-11-15 11:10:19,419][WARN ][cluster.service          ] [Desmond Pitt] failed to reconnect to node [Desmond Pitt][2-uc6WDiRwSTwsSXf8jg6A][bitnami-nginxstack-50be][inet[/100.114.244.37:9300]]
org.elasticsearch.transport.ConnectTransportException: [Desmond Pitt][inet[/100.114.244.37:9300]] connect_timeout[30s]
	at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:807)
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:741)
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:714)
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:150)
	at org.elasticsearch.cluster.service.InternalClusterService$ReconnectToNodes.run(InternalClusterService.java:539)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: /100.114.244.37:9300
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
 ...
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	... 3 more