OutOfMemory Exception on client Node

VB1 · May 27, 2014, 9:03pm

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did not
find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}, added
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from
master
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

VB1 · May 28, 2014, 5:39pm

Can anyone please help us here?

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}, added
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from
master
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · May 28, 2014, 10:25pm

What java version are you using? How big are these indexes? How many nodes,
and number of each type do you have?

I'll state the obvious and mention that 0.90.N is really old now and you'd
do well to upgrade to at least 1.0.N.

But looking at your logs I can see this - failed to connect to node
[[ELS-10.76.121.130]. Is this the node that is dropping off due to OOM?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 29 May 2014 03:39, VB vishal.batghare@gmail.com wrote:

Can anyone please help us here?

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:647)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:615)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:129)
at org.elasticsearch.cluster.service.InternalClusterService$
UpdateTask.run(InternalClusterService.java:396)
at org.elasticsearch.common.util.concurrent.
PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(
PrioritizedEsThreadPoolExecutor.java:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [33.2mb]->[54.7mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [54.7mb]->[65.5mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [65.5mb]->[66.4mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN ][netty.channel.socket.nio.AbstractNioSelector]
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(
WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.
SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Yo0CRss%3DVO0cZ5Mgx9hmzB76%3DDWJJ1%2BvmWm_YMjWvWOA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

VB1 · May 28, 2014, 10:46pm

Thanks mark for the reply.

We are using 90.11 of Es and java 7.45. Node which is throwing OOM
is BUS9364B62 which is client node.

Indexes are very big. Each index has around 150gb of data,

We cannot move to newer version now as we are working of 90.11 from quite
some time.

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}, added
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from
master
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · May 28, 2014, 10:58pm

How many nodes, and of those which are data, master and client?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 29 May 2014 08:46, VB vishal.batghare@gmail.com wrote:

Thanks mark for the reply.

We are using 90.11 of Es and java 7.45. Node which is throwing OOM
is BUS9364B62 which is client node.

Indexes are very big. Each index has around 150gb of data,

We cannot move to newer version now as we are working of 90.11 from quite
some time.

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:647)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:615)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:129)
at org.elasticsearch.cluster.service.InternalClusterService$
UpdateTask.run(InternalClusterService.java:396)
at org.elasticsearch.common.util.concurrent.
PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(
PrioritizedEsThreadPoolExecutor.java:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [33.2mb]->[54.7mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [54.7mb]->[65.5mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [65.5mb]->[66.4mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN ][netty.channel.socket.nio.AbstractNioSelector]
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(
WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.
SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aL%3DsQ9HQJoTkhS4hNdmuzmNe69ovxpNAjO3N5z51FPTw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

VB1 · May 28, 2014, 11:13pm

around 10 clients with 10gb committed RAM

3 masters with 2gb committed RAM

32 data nodes with 18gb committed RAM.

On Wednesday, 28 May 2014 15:59:23 UTC-7, Mark Walkom wrote:

How many nodes, and of those which are data, master and client?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 29 May 2014 08:46, VB <vishal....@gmail.com <javascript:>> wrote:

Thanks mark for the reply.

We are using 90.11 of Es and java 7.45. Node which is throwing OOM
is BUS9364B62 which is client node.

Indexes are very big. Each index has around 150gb of data,

We cannot move to newer version now as we are working of 90.11 from quite
some time.

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:647)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:615)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:129)
at org.elasticsearch.cluster.service.InternalClusterService$
UpdateTask.run(InternalClusterService.java:396)
at org.elasticsearch.common.util.concurrent.
PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(
PrioritizedEsThreadPoolExecutor.java:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN ][netty.channel.socket.nio.AbstractNioSelector]
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(
WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.
SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1e8f4ca1-0947-43f7-82e9-53d5d8da82db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

error804 · May 29, 2014, 4:47am

i think its time to restart the cluster. If you are getting this problem
frequently you have to add some nodes to that cluster

On Wed, May 28, 2014 at 2:48 AM, VB vishal.batghare@gmail.com wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}, added
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from
master [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMUueYkwFKM76vgrRgBakzLYT-KBZh6J_K-KYU7ikF%2B4zLLCtw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

VB1 · May 29, 2014, 5:18pm

These exceptions are happening on client node and not on data node. I do
not think we need any restart of cluster.

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}, added
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from
master
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · May 29, 2014, 10:16pm

Then you need to add more nodes or more RAM to existing nodes or delete
some data, essentially you are hitting the limits of what you can add to
the cluster.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 30 May 2014 03:18, VB vishal.batghare@gmail.com wrote:

These exceptions are happening on client node and not on data node. I do
not think we need any restart of cluster.

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node [[ELS-10.76.121.130][
BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(
NettyTransport.java:727)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:647)
at org.elasticsearch.transport.netty.NettyTransport.
connectToNode(NettyTransport.java:615)
at org.elasticsearch.transport.TransportService.connectToNode(
TransportService.java:129)
at org.elasticsearch.cluster.service.InternalClusterService$
UpdateTask.run(InternalClusterService.java:396)
at org.elasticsearch.common.util.concurrent.
PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(
PrioritizedEsThreadPoolExecutor.java:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [33.2mb]->[54.7mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/
10.76.121.130:9300]]{data=false, max_local_storage_nodes=1,
master=true}], reason [failed to perform initial connect
[[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [54.7mb]->[65.5mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [65.5mb]->[66.4mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN ][netty.channel.socket.nio.AbstractNioSelector]
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(
WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.elasticsearch.common.netty.channel.socket.nio.
SelectorUtil.select(SelectorUtil.java:68)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bqJz5HVzL96vNTi1x2-Eykwy3tnxdGNpeW%3Dxyz9T9wKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

VB1 · June 4, 2014, 6:40pm

These are more log statements after GC.

[2014-06-04 14:47:12,939][INFO ][cluster.service ] [BUS2F2801F3]
master {new
[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true}, previous
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}}, removed
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-master_failed
([ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true})
[2014-06-04 14:48:03,969][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55503][489] duration [49.6s], collections [1]/[49.9s], total
[49.6s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[51.3mb]->[42.8mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:48:40,256][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55504][490] duration [35.7s], collections [1]/[36.2s], total
[35.7s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[42.8mb]->[58.6mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:49:30,335][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55505][491] duration [49.9s], collections [1]/[50s], total
[49.9s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[58.6mb]->[63.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:49:30,350][INFO ][discovery.zen ] [BUS2F2801F3]
master_left
[[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to ping, tried [3]
times, each with maximum [30s] timeout]
[2014-06-04 14:49:30,865][WARN ][discovery.zen ] [BUS2F2801F3]
not enough master nodes after master left (reason = failed to ping, tried
[3] times, each with maximum [30s] timeout), current nodes:
{[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5_j-Q][inet[/10.76.122.38:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][inet[/10.76.125.48:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300]]{client=true,
data=false,
master=false},[ELS-10.76.120.62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[BUS9364B62][YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][inet[/10.76.125.38:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.214][H8N9nIU0TKyGv_prKyRVCQ][inet[/10.76.124.214:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/10.76.120.87:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][inet[/10.76.122.67:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][inet[/10.76.121.129:9300]]{data=false,
max_local_storage_nodes=1,
master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-_AXw][inet[/10.76.122.20:9300]]{client=true,
data=false,
master=false},[EDI1AE4FD76][UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][inet[/10.76.125.46:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/10.76.120.82:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.34][kPSYm9G8R8i_z2skK_jq1g][inet[/10.76.122.34:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][inet[/10.76.125.43:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/10.76.124.227:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][inet[/10.76.124.225:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.61][_60f636_QsOPIWN0tKyN2A][inet[/10.76.120.61:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/10.76.124.18:9300]]{client=true,
data=false,
master=false},[BUS936E1B3][Vnr_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][inet[/10.76.121.186:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EC098][SvH6eHsHRoyz8_PEig46LA][inet[/10.76.120.83:9300]]{client=true,
data=false,
master=false},[ELS-10.76.122.39][swxUZAjCTCeBZS2-Uo_Bpw][inet[/10.76.122.39:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][inet[/10.76.120.63:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.37][TBw4vykLR_uRT76gt23UUQ][inet[/10.76.122.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og_DdvQ][inet[/10.76.122.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi_iA][inet[/10.76.120.86:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.50][jv7xpDnpTiOo21Kg_JgkVw][inet[/10.76.125.50:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][inet[/10.76.124.223:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/10.76.120.84:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.221][nB65C4WGRiSCntVZoovX_Q][inet[/10.76.124.221:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][inet[/10.76.124.224:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300]]{client=true,
data=false,
master=false},[STG0A4C78E2][ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true,
data=false,
master=false},[BUS2F2801F3][al8pR84OSlOIhnpT0JSJHQ][inet[BUS2F2801F3.SWLAB.RMSCLOUD.NET/10.76.122.123:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][inet[/10.76.124.218:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][inet[/10.76.125.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/10.76.121.248:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][inet[/10.76.125.49:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][inet[/10.76.122.35:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/10.76.120.85:9300]]{client=true,
data=false,
master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true,
data=false, master=false},}
[2014-06-04 14:50:14,937][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55506][492] duration [44.5s], collections [1]/[44.6s], total
[44.5s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[63.7mb]->[65.9mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 14:50:14,937][INFO ][cluster.service ] [BUS2F2801F3]
removed
{[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5_j-Q][inet[/10.76.122.38:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][inet[/10.76.125.48:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300]]{client=true,
data=false,
master=false},[ELS-10.76.120.62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[BUS9364B62][YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][inet[/10.76.125.38:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.214][H8N9nIU0TKyGv_prKyRVCQ][inet[/10.76.124.214:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/10.76.120.87:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][inet[/10.76.122.67:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][inet[/10.76.121.129:9300]]{data=false,
max_local_storage_nodes=1,
master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-_AXw][inet[/10.76.122.20:9300]]{client=true,
data=false,
master=false},[EDI1AE4FD76][UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][inet[/10.76.125.46:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/10.76.120.82:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.34][kPSYm9G8R8i_z2skK_jq1g][inet[/10.76.122.34:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][inet[/10.76.125.43:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/10.76.124.227:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][inet[/10.76.124.225:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.61][_60f636_QsOPIWN0tKyN2A][inet[/10.76.120.61:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max_local_storage_nodes=1,
master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/10.76.124.18:9300]]{client=true,
data=false,
master=false},[BUS936E1B3][Vnr_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][inet[/10.76.121.186:9300]]{max_local_storage_nodes=1,
master=false},[EDI1A1EC098][SvH6eHsHRoyz8_PEig46LA][inet[/10.76.120.83:9300]]{client=true,
data=false,
master=false},[ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1,
master=true},[ELS-10.76.122.39][swxUZAjCTCeBZS2-Uo_Bpw][inet[/10.76.122.39:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][inet[/10.76.120.63:9300]]{client=true,
data=false, max_local_storage_nodes=1,
master=false},[ELS-10.76.125.45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.37][TBw4vykLR_uRT76gt23UUQ][inet[/10.76.122.37:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og_DdvQ][inet[/10.76.122.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi_iA][inet[/10.76.120.86:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.50][jv7xpDnpTiOo21Kg_JgkVw][inet[/10.76.125.50:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][inet[/10.76.124.223:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/10.76.120.84:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.221][nB65C4WGRiSCntVZoovX_Q][inet[/10.76.124.221:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][inet[/10.76.124.224:9300]]{max_local_storage_nodes=1,
master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300]]{client=true,
data=false,
master=false},[STG0A4C78E2][ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true,
data=false,
master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][inet[/10.76.124.218:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][inet[/10.76.125.36:9300]]{max_local_storage_nodes=1,
master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/10.76.121.248:9300]]{client=true,
data=false,
master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][inet[/10.76.125.49:9300]]{max_local_storage_nodes=1,
master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][inet[/10.76.122.35:9300]]{max_local_storage_nodes=1,
master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/10.76.120.85:9300]]{client=true,
data=false,
master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true,
data=false, master=false},}, reason: zen-disco-master_failed
([ELS-10.76.121.131][dg_r12_nQbqIT_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false,
max_local_storage_nodes=1, master=true})
[2014-06-04 14:52:25,048][WARN ][monitor.jvm ] [BUS2F2801F3]
[gc][old][55507][493] duration [43.7s], collections [1]/[43.7s], total
[43.7s]/[4.5h], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.9mb]->[66.3mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-06-04 15:07:08,535][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
[2014-06-04 15:07:08,535][DEBUG][action.search.type ] [BUS2F2801F3]
[1004_exposureindex][49], node[ot8ch3V6TYGKOLYnCY9cYw], [R], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@14495084]
lastShard [true]
org.elasticsearch.transport.SendRequestTransportException:
[ELS-10.76.125.44][inet[/10.76.125.44:9300]][search/phase/query]
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:202)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:173)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:208)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.transport.NodeNotConnectedException:
[ELS-10.76.125.44][inet[/10.76.125.44:9300]] Node not connected
at
org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:859)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:540)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:189)
... 8 more

On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote:

Hi all,

We are running 90.11 and we have a cluster with client, master and data
nodes.

Our client nodes are using dedicated 10g memory.

But we are seeing these outofmemory exceptions.

I tried to correlate this log time with logs in our exception but I did
not find any query which we could be causing this issue.

Our cluster has 37 indexes with 50 shards and 1 replica.

Some indexes has data and some doesn't.

[2014-05-27 16:26:34,688][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327409][8] duration [33s], collections [1]/[33.5s], total
[33s]/[3.8m], memory [9gb]->[9.4gb]/[9.9gb], all_pools {[young]
[520.2kb]->[112.3mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:27:19,992][INFO ][cluster.service ] [BUS9364B62]
detected_master
[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}, added
{[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true},}, reason: zen-disco-receive(from
master
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}])
[2014-05-27 16:27:20,008][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:27:20,008][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total
[44.3s]/[4.6m], memory [9.4gb]->[9.8gb]/[9.9gb], all_pools {[young]
[112.3mb]->[475mb]/[532.5mb]}{[survivor] [0b]->[0b]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:06,856][WARN ][cluster.service ] [BUS9364B62]
failed to connect to node
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}]
org.elasticsearch.transport.ConnectTransportException:
[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect_timeout[30s]
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-05-27 16:28:06,856][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total
[45.3s]/[5.3m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor] [41mb]->[33.2mb]/[66.5mb]}{[old]
[9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:28:53,876][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327413][11] duration [46.4s], collections [1]/[47s], total
[46.4s]/[6.1m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[33.2mb]->[54.7mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:29:40,990][INFO ][discovery.zen ] [BUS9364B62]
master_left
[[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false,
max_local_storage_nodes=1, master=true}], reason [failed to perform initial
connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]]
connect_timeout[30s]]]
[2014-05-27 16:29:40,990][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total
[46.8s]/[6.9m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[54.7mb]->[65.5mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:30:27,589][WARN ][monitor.jvm ] [BUS9364B62]
[gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total
[46.5s]/[7.7m], memory [9.9gb]->[9.9gb]/[9.9gb], all_pools {[young]
[532.5mb]->[532.5mb]/[532.5mb]}{[survivor]
[65.5mb]->[66.4mb]/[66.5mb]}{[old] [9.3gb]->[9.3gb]/[9.3gb]}
[2014-05-27 16:48:27,804][WARN
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the
selector loop.
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.iterator(ArrayList.java:814)
at
sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a73f803b-1009-4bb4-9ce8-1693a646a82e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Out of memory on Client Node Elasticsearch	1	456	April 21, 2020
Elasticsearch showing NodeDisconnectedException followed by Outofmemory? Elasticsearch	3	1124	July 25, 2017
Out of memory of data nodes Elasticsearch	5	1243	February 23, 2018
OutOfMemoryException:Client node Elasticsearch	1	618	April 21, 2017
Clarification in Elasticsearch OUTOFMEMORY warning? Elasticsearch	2	675	July 21, 2017

OutOfMemory Exception on client Node

Related Topics