# OutOfMemory Exception on client Node **URL:** \ **Category:** Elasticsearch\ **Created:** [May 27, 2014, 9:03pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764 "2014-05-27T21:03:29Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![VB1](https://avatars.discourse-cdn.com/v4/letter/v/2acd7d/32.png) [@VB1](https://discuss.elastic.co/u/VB1)\ **Post date:** [May 27, 2014, 9:03pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/1 "2014-05-27T21:03:29Z")
Hi all, We are running 90.11 and we have a cluster with client, master and data nodes. Our client nodes are using dedicated 10g memory. But we are seeing these outofmemory exceptions. I tried to correlate this log time with logs in our exception but I did not find any query which we could be causing this issue. Our cluster has 37 indexes with 50 shards and 1 replica. Some indexes has data and some doesn't. [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] [9gb]-\>[9.3gb]/[9.3gb]} [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] detected\_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}]) [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] failed to connect to node [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}] org.elasticsearch.transport.ConnectTransportException: [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727) at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647) at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615) at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129) at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-05-27 16:48:27,804][WARN ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.iterator(ArrayList.java:814) at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![VB1](https://avatars.discourse-cdn.com/v4/letter/v/2acd7d/32.png) [@VB1](https://discuss.elastic.co/u/VB1)\ **Post date:** [May 28, 2014, 5:39pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/2 "2014-05-28T17:39:58Z")
Can anyone please help us here? On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > Hi all, > > We are running 90.11 and we have a cluster with client, master and data > nodes. > > Our client nodes are using dedicated 10g memory. > > But we are seeing these outofmemory exceptions. > > I tried to correlate this log time with logs in our exception but I did > not find any query which we could be causing this issue. > > Our cluster has 37 indexes with 50 shards and 1 replica. > > Some indexes has data and some doesn't. > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > detected\_master > [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}, added > {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from > master > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}]) > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > failed to connect to node > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}] > org.elasticsearch.transport.ConnectTransportException: > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615) > at > org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129) > at > org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396) > at > org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:48:27,804][WARN > ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the > selector loop. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.iterator(ArrayList.java:814) > at > sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2014, 10:25pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/3 "2014-05-28T22:25:03Z")
What java version are you using? How big are these indexes? How many nodes, and number of each type do you have? I'll state the obvious and mention that 0.90.N is really old now and you'd do well to upgrade to at least 1.0.N. But looking at your logs I can see this - failed to connect to node [[ELS-10.76.121.130]. Is this the node that is dropping off due to OOM? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 29 May 2014 03:39, VB [vishal.batghare@gmail.com](mailto:vishal.batghare@gmail.com) wrote: > Can anyone please help us here? > > On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > > > Hi all, > > > > We are running 90.11 and we have a cluster with client, master and data > > nodes. > > > > Our client nodes are using dedicated 10g memory. > > > > But we are seeing these outofmemory exceptions. > > > > I tried to correlate this log time with logs in our exception but I did > > not find any query which we could be causing this issue. > > > > Our cluster has 37 indexes with 50 shards and 1 replica. > > > > Some indexes has data and some doesn't. > > > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > [9gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > > detected\_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][ > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > max\_local\_storage\_nodes=1, master=true}]) > > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}], reason [failed to perform initial connect > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > > failed to connect to node [[ELS-10.76.121.130][ > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > max\_local\_storage\_nodes=1, master=true}] > > org.elasticsearch.transport.ConnectTransportException: > > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > > at org.elasticsearch.transport.netty.NettyTransport.connectToChannels( > > NettyTransport.java:727) > > at org.elasticsearch.transport.netty.NettyTransport. > > connectToNode(NettyTransport.java:647) > > at org.elasticsearch.transport.netty.NettyTransport. > > connectToNode(NettyTransport.java:615) > > at org.elasticsearch.transport.TransportService.connectToNode( > > TransportService.java:129) > > at org.elasticsearch.cluster.service.InternalClusterService$ > > UpdateTask.run(InternalClusterService.java:396) > > at org.elasticsearch.common.util.concurrent. > > PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run( > > PrioritizedEsThreadPoolExecutor.java:135) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}], reason [failed to perform initial connect > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:48:27,804][WARN][netty.channel.socket.nio.AbstractNioSelector] > > Unexpected exception in the selector loop. > > java.lang.OutOfMemoryError: Java heap space > > at java.util.ArrayList.iterator(ArrayList.java:814) > > at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys( > > WindowsSelectorImpl.java:496) > > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > > at org.elasticsearch.common.netty.channel.socket.nio. > > SelectorUtil.select(SelectorUtil.java:68) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.select(AbstractNioSelector.java:415) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.run(AbstractNioSelector.java:212) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioWorker.run(AbstractNioWorker.java:89) > > at org.elasticsearch.common.netty.channel.socket.nio. > > NioWorker.run(NioWorker.java:178) > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > > ThreadRenamingRunnable.java:108) > > at org.elasticsearch.common.netty.util.internal. > > DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8d91abba-56e0-4f40-bb05-eb663be7e154%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yo0CRss%3DVO0cZ5Mgx9hmzB76%3DDWJJ1%2BvmWm\_YMjWvWOA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yo0CRss%3DVO0cZ5Mgx9hmzB76%3DDWJJ1%2BvmWm_YMjWvWOA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![VB1](https://avatars.discourse-cdn.com/v4/letter/v/2acd7d/32.png) [@VB1](https://discuss.elastic.co/u/VB1)\ **Post date:** [May 28, 2014, 10:46pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/4 "2014-05-28T22:46:41Z")
Thanks mark for the reply. We are using 90.11 of Es and java 7.45. Node which is throwing OOM is BUS9364B62 which is client node. Indexes are very big. Each index has around 150gb of data, We cannot move to newer version now as we are working of 90.11 from quite some time. On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > Hi all, > > We are running 90.11 and we have a cluster with client, master and data > nodes. > > Our client nodes are using dedicated 10g memory. > > But we are seeing these outofmemory exceptions. > > I tried to correlate this log time with logs in our exception but I did > not find any query which we could be causing this issue. > > Our cluster has 37 indexes with 50 shards and 1 replica. > > Some indexes has data and some doesn't. > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > detected\_master > [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}, added > {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from > master > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}]) > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > failed to connect to node > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}] > org.elasticsearch.transport.ConnectTransportException: > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615) > at > org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129) > at > org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396) > at > org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:48:27,804][WARN > ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the > selector loop. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.iterator(ArrayList.java:814) > at > sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2014, 10:58pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/5 "2014-05-28T22:58:52Z")
How many nodes, and of those which are data, master and client? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 29 May 2014 08:46, VB [vishal.batghare@gmail.com](mailto:vishal.batghare@gmail.com) wrote: > Thanks mark for the reply. > > We are using 90.11 of Es and java 7.45. Node which is throwing OOM > is BUS9364B62 which is client node. > > Indexes are very big. Each index has around 150gb of data, > > We cannot move to newer version now as we are working of 90.11 from quite > some time. > > On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > > > Hi all, > > > > We are running 90.11 and we have a cluster with client, master and data > > nodes. > > > > Our client nodes are using dedicated 10g memory. > > > > But we are seeing these outofmemory exceptions. > > > > I tried to correlate this log time with logs in our exception but I did > > not find any query which we could be causing this issue. > > > > Our cluster has 37 indexes with 50 shards and 1 replica. > > > > Some indexes has data and some doesn't. > > > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > [9gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > > detected\_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][ > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > max\_local\_storage\_nodes=1, master=true}]) > > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}], reason [failed to perform initial connect > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > > failed to connect to node [[ELS-10.76.121.130][ > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > max\_local\_storage\_nodes=1, master=true}] > > org.elasticsearch.transport.ConnectTransportException: > > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > > at org.elasticsearch.transport.netty.NettyTransport.connectToChannels( > > NettyTransport.java:727) > > at org.elasticsearch.transport.netty.NettyTransport. > > connectToNode(NettyTransport.java:647) > > at org.elasticsearch.transport.netty.NettyTransport. > > connectToNode(NettyTransport.java:615) > > at org.elasticsearch.transport.TransportService.connectToNode( > > TransportService.java:129) > > at org.elasticsearch.cluster.service.InternalClusterService$ > > UpdateTask.run(InternalClusterService.java:396) > > at org.elasticsearch.common.util.concurrent. > > PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run( > > PrioritizedEsThreadPoolExecutor.java:135) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}], reason [failed to perform initial connect > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:48:27,804][WARN][netty.channel.socket.nio.AbstractNioSelector] > > Unexpected exception in the selector loop. > > java.lang.OutOfMemoryError: Java heap space > > at java.util.ArrayList.iterator(ArrayList.java:814) > > at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys( > > WindowsSelectorImpl.java:496) > > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > > at org.elasticsearch.common.netty.channel.socket.nio. > > SelectorUtil.select(SelectorUtil.java:68) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.select(AbstractNioSelector.java:415) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.run(AbstractNioSelector.java:212) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioWorker.run(AbstractNioWorker.java:89) > > at org.elasticsearch.common.netty.channel.socket.nio. > > NioWorker.run(NioWorker.java:178) > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > > ThreadRenamingRunnable.java:108) > > at org.elasticsearch.common.netty.util.internal. > > DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aL%3DsQ9HQJoTkhS4hNdmuzmNe69ovxpNAjO3N5z51FPTw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aL%3DsQ9HQJoTkhS4hNdmuzmNe69ovxpNAjO3N5z51FPTw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![VB1](https://avatars.discourse-cdn.com/v4/letter/v/2acd7d/32.png) [@VB1](https://discuss.elastic.co/u/VB1)\ **Post date:** [May 28, 2014, 11:13pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/6 "2014-05-28T23:13:21Z")
around 10 clients with 10gb committed RAM 3 masters with 2gb committed RAM 32 data nodes with 18gb committed RAM. On Wednesday, 28 May 2014 15:59:23 UTC-7, Mark Walkom wrote: > How many nodes, and of those which are data, master and client? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 29 May 2014 08:46, VB \<[vishal....@gmail.com](mailto:vishal....@gmail.com) \\> wrote: > > > Thanks mark for the reply. > > > > We are using 90.11 of Es and java 7.45. Node which is throwing OOM > > is BUS9364B62 which is client node. > > > > Indexes are very big. Each index has around 150gb of data, > > > > We cannot move to newer version now as we are working of 90.11 from quite > > some time. > > > > On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > > > > > Hi all, > > > > > > We are running 90.11 and we have a cluster with client, master and data > > > nodes. > > > > > > Our client nodes are using dedicated 10g memory. > > > > > > But we are seeing these outofmemory exceptions. > > > > > > I tried to correlate this log time with logs in our exception but I did > > > not find any query which we could be causing this issue. > > > > > > Our cluster has 37 indexes with 50 shards and 1 replica. > > > > > > Some indexes has data and some doesn't. > > > > > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > > > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > > > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > > > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > > [9gb]-\>[9.3gb]/[9.3gb]} > > > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > > > detected\_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > > master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > > master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][ > > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > > max\_local\_storage\_nodes=1, master=true}]) > > > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > > master=true}], reason [failed to perform initial connect > > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > > > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > > > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > > > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > > > failed to connect to node [[ELS-10.76.121.130][ > > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > > max\_local\_storage\_nodes=1, master=true}] > > > org.elasticsearch.transport.ConnectTransportException: > > > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > > > at org.elasticsearch.transport.netty.NettyTransport.connectToChannels( > > > NettyTransport.java:727) > > > at org.elasticsearch.transport.netty.NettyTransport. > > > connectToNode(NettyTransport.java:647) > > > at org.elasticsearch.transport.netty.NettyTransport. > > > connectToNode(NettyTransport.java:615) > > > at org.elasticsearch.transport.TransportService.connectToNode( > > > TransportService.java:129) > > > at org.elasticsearch.cluster.service.InternalClusterService$ > > > UpdateTask.run(InternalClusterService.java:396) > > > at org.elasticsearch.common.util.concurrent. > > > PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run( > > > PrioritizedEsThreadPoolExecutor.java:135) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1145) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:744) > > > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > > > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > > > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > > > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > > > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > > > [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > > > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > > master=true}], reason [failed to perform initial connect > > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > > > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > > > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > > > [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > > > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > > > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > > > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > > > [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > > > [2014-05-27 16:48:27,804][WARN][netty.channel.socket.nio.AbstractNioSelector] > > > Unexpected exception in the selector loop. > > > java.lang.OutOfMemoryError: Java heap space > > > at java.util.ArrayList.iterator(ArrayList.java:814) > > > at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys( > > > WindowsSelectorImpl.java:496) > > > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > SelectorUtil.select(SelectorUtil.java:68) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > AbstractNioSelector.select(AbstractNioSelector.java:415) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > AbstractNioSelector.run(AbstractNioSelector.java:212) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > AbstractNioWorker.run(AbstractNioWorker.java:89) > > > at org.elasticsearch.common.netty.channel.socket.nio. > > > NioWorker.run(NioWorker.java:178) > > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > > > ThreadRenamingRunnable.java:108) > > > at org.elasticsearch.common.netty.util.internal. > > > DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1145) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:744) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6d5d9db-7a2c-4644-9df4-934b2c1405e4%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1e8f4ca1-0947-43f7-82e9-53d5d8da82db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1e8f4ca1-0947-43f7-82e9-53d5d8da82db%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![error804](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@error804](https://discuss.elastic.co/u/error804)\ **Post date:** [May 29, 2014, 4:47am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/7 "2014-05-29T04:47:59Z")
i think its time to restart the cluster. If you are getting this problem frequently you have to add some nodes to that cluster On Wed, May 28, 2014 at 2:48 AM, VB [vishal.batghare@gmail.com](mailto:vishal.batghare@gmail.com) wrote: > Hi all, > > We are running 90.11 and we have a cluster with client, master and data > nodes. > > Our client nodes are using dedicated 10g memory. > > But we are seeing these outofmemory exceptions. > > I tried to correlate this log time with logs in our exception but I did > not find any query which we could be causing this issue. > > Our cluster has 37 indexes with 50 shards and 1 replica. > > Some indexes has data and some doesn't. > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > detected\_master > [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}, added > {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from > master [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}]) > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > failed to connect to node > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}] > org.elasticsearch.transport.ConnectTransportException: > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615) > at > org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129) > at > org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396) > at > org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:48:27,804][WARN > ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the > selector loop. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.iterator(ArrayList.java:814) > at > sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94ed1a9c-9036-4f68-98b4-ccad1be91274%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMUueYkwFKM76vgrRgBakzLYT-KBZh6J\_K-KYU7ikF%2B4zLLCtw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMUueYkwFKM76vgrRgBakzLYT-KBZh6J_K-KYU7ikF%2B4zLLCtw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![VB1](https://avatars.discourse-cdn.com/v4/letter/v/2acd7d/32.png) [@VB1](https://discuss.elastic.co/u/VB1)\ **Post date:** [May 29, 2014, 5:18pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/8 "2014-05-29T17:18:10Z")
These exceptions are happening on client node and not on data node. I do not think we need any restart of cluster. On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > Hi all, > > We are running 90.11 and we have a cluster with client, master and data > nodes. > > Our client nodes are using dedicated 10g memory. > > But we are seeing these outofmemory exceptions. > > I tried to correlate this log time with logs in our exception but I did > not find any query which we could be causing this issue. > > Our cluster has 37 indexes with 50 shards and 1 replica. > > Some indexes has data and some doesn't. > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > detected\_master > [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}, added > {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from > master > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}]) > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > failed to connect to node > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}] > org.elasticsearch.transport.ConnectTransportException: > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615) > at > org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129) > at > org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396) > at > org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:48:27,804][WARN > ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the > selector loop. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.iterator(ArrayList.java:814) > at > sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 29, 2014, 10:16pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/9 "2014-05-29T22:16:50Z")
Then you need to add more nodes or more RAM to existing nodes or delete some data, essentially you are hitting the limits of what you can add to the cluster. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 30 May 2014 03:18, VB [vishal.batghare@gmail.com](mailto:vishal.batghare@gmail.com) wrote: > These exceptions are happening on client node and not on data node. I do > not think we need any restart of cluster. > > On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > > > Hi all, > > > > We are running 90.11 and we have a cluster with client, master and data > > nodes. > > > > Our client nodes are using dedicated 10g memory. > > > > But we are seeing these outofmemory exceptions. > > > > I tried to correlate this log time with logs in our exception but I did > > not find any query which we could be causing this issue. > > > > Our cluster has 37 indexes with 50 shards and 1 replica. > > > > Some indexes has data and some doesn't. > > > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > [9gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > > detected\_master [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}, added {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true},}, reason: zen-disco-receive(from master [[ELS-10.76.121.130][ > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > max\_local\_storage\_nodes=1, master=true}]) > > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}], reason [failed to perform initial connect > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > > failed to connect to node [[ELS-10.76.121.130][ > > BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > > max\_local\_storage\_nodes=1, master=true}] > > org.elasticsearch.transport.ConnectTransportException: > > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > > at org.elasticsearch.transport.netty.NettyTransport.connectToChannels( > > NettyTransport.java:727) > > at org.elasticsearch.transport.netty.NettyTransport. > > connectToNode(NettyTransport.java:647) > > at org.elasticsearch.transport.netty.NettyTransport. > > connectToNode(NettyTransport.java:615) > > at org.elasticsearch.transport.TransportService.connectToNode( > > TransportService.java:129) > > at org.elasticsearch.cluster.service.InternalClusterService$ > > UpdateTask.run(InternalClusterService.java:396) > > at org.elasticsearch.common.util.concurrent. > > PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run( > > PrioritizedEsThreadPoolExecutor.java:135) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > > master\_left [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/ > > 10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, > > master=true}], reason [failed to perform initial connect > > [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s]]] > > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] > > [9.3gb]-\>[9.3gb]/[9.3gb]} > > [2014-05-27 16:48:27,804][WARN][netty.channel.socket.nio.AbstractNioSelector] > > Unexpected exception in the selector loop. > > java.lang.OutOfMemoryError: Java heap space > > at java.util.ArrayList.iterator(ArrayList.java:814) > > at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys( > > WindowsSelectorImpl.java:496) > > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > > at org.elasticsearch.common.netty.channel.socket.nio. > > SelectorUtil.select(SelectorUtil.java:68) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.select(AbstractNioSelector.java:415) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioSelector.run(AbstractNioSelector.java:212) > > at org.elasticsearch.common.netty.channel.socket.nio. > > AbstractNioWorker.run(AbstractNioWorker.java:89) > > at org.elasticsearch.common.netty.channel.socket.nio. > > NioWorker.run(NioWorker.java:178) > > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > > ThreadRenamingRunnable.java:108) > > at org.elasticsearch.common.netty.util.internal. > > DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1145) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a43d791c-ebce-45b3-b345-52c8196ee4c4%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bqJz5HVzL96vNTi1x2-Eykwy3tnxdGNpeW%3Dxyz9T9wKQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bqJz5HVzL96vNTi1x2-Eykwy3tnxdGNpeW%3Dxyz9T9wKQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![VB1](https://avatars.discourse-cdn.com/v4/letter/v/2acd7d/32.png) [@VB1](https://discuss.elastic.co/u/VB1)\ **Post date:** [June 4, 2014, 6:40pm UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/10 "2014-06-04T18:40:59Z")
These are more log statements after GC. [2014-06-04 14:47:12,939][INFO][cluster.service] [BUS2F2801F3] master {new [ELS-10.76.121.131][dg\_r12\_nQbqIT\_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}, previous [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}}, removed {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-master\_failed ([ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}) [2014-06-04 14:48:03,969][WARN][monitor.jvm] [BUS2F2801F3] [gc][old][55503][489] duration [49.6s], collections [1]/[49.9s], total [49.6s]/[4.5h], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [51.3mb]-\>[42.8mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-06-04 14:48:40,256][WARN][monitor.jvm] [BUS2F2801F3] [gc][old][55504][490] duration [35.7s], collections [1]/[36.2s], total [35.7s]/[4.5h], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [42.8mb]-\>[58.6mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-06-04 14:49:30,335][WARN][monitor.jvm] [BUS2F2801F3] [gc][old][55505][491] duration [49.9s], collections [1]/[50s], total [49.9s]/[4.5h], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [58.6mb]-\>[63.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-06-04 14:49:30,350][INFO][discovery.zen] [BUS2F2801F3] master\_left [[ELS-10.76.121.131][dg\_r12\_nQbqIT\_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}], reason [failed to ping, tried [3] times, each with maximum [30s] timeout] [2014-06-04 14:49:30,865][WARN][discovery.zen] [BUS2F2801F3] not enough master nodes after master left (reason = failed to ping, tried [3] times, each with maximum [30s] timeout), current nodes: {[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5\_j-Q][inet[/10.76.122.38:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][inet[/10.76.125.48:9300]]{max\_local\_storage\_nodes=1, master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300]]{client=true, data=false, master=false},[ELS-10.76.120.62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true, data=false, max\_local\_storage\_nodes=1, master=false},[BUS9364B62][YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true, data=false, master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][inet[/10.76.125.38:9300]]{max\_local\_storage\_nodes=1, master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300]]{client=true, data=false, master=false},[ELS-10.76.124.214][H8N9nIU0TKyGv\_prKyRVCQ][inet[/10.76.124.214:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/10.76.120.87:9300]]{client=true, data=false, master=false},[ELS-10.76.125.40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][inet[/10.76.122.67:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][inet[/10.76.121.129:9300]]{data=false, max\_local\_storage\_nodes=1, master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-\_AXw][inet[/10.76.122.20:9300]]{client=true, data=false, master=false},[EDI1AE4FD76][UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true, data=false, master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][inet[/10.76.125.46:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/10.76.120.82:9300]]{client=true, data=false, master=false},[ELS-10.76.121.188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.34][kPSYm9G8R8i\_z2skK\_jq1g][inet[/10.76.122.34:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][inet[/10.76.125.43:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/10.76.124.227:9300]]{client=true, data=false, master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][inet[/10.76.124.225:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.120.61][\_60f636\_QsOPIWN0tKyN2A][inet[/10.76.120.61:9300]]{client=true, data=false, max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/10.76.124.18:9300]]{client=true, data=false, master=false},[BUS936E1B3][Vnr\_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true, data=false, master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][inet[/10.76.121.186:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1A1EC098][SvH6eHsHRoyz8\_PEig46LA][inet[/10.76.120.83:9300]]{client=true, data=false, master=false},[ELS-10.76.122.39][swxUZAjCTCeBZS2-Uo\_Bpw][inet[/10.76.122.39:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][inet[/10.76.120.63:9300]]{client=true, data=false, max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.37][TBw4vykLR\_uRT76gt23UUQ][inet[/10.76.122.37:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og\_DdvQ][inet[/10.76.122.36:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi\_iA][inet[/10.76.120.86:9300]]{client=true, data=false, master=false},[ELS-10.76.125.50][jv7xpDnpTiOo21Kg\_JgkVw][inet[/10.76.125.50:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][inet[/10.76.124.223:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/10.76.120.84:9300]]{client=true, data=false, master=false},[ELS-10.76.124.221][nB65C4WGRiSCntVZoovX\_Q][inet[/10.76.124.221:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][inet[/10.76.124.224:9300]]{max\_local\_storage\_nodes=1, master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300]]{client=true, data=false, master=false},[STG0A4C78E2][ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true, data=false, master=false},[BUS2F2801F3][al8pR84OSlOIhnpT0JSJHQ][inet[[BUS2F2801F3.SWLAB.RMSCLOUD.NET/10.76.122.123:9300](http://BUS2F2801F3.SWLAB.RMSCLOUD.NET/10.76.122.123:9300)]]{client=true, data=false, master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][inet[/10.76.124.218:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][inet[/10.76.125.36:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/10.76.121.248:9300]]{client=true, data=false, master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][inet[/10.76.125.49:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][inet[/10.76.122.35:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/10.76.120.85:9300]]{client=true, data=false, master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true, data=false, master=false},} [2014-06-04 14:50:14,937][WARN][monitor.jvm] [BUS2F2801F3] [gc][old][55506][492] duration [44.5s], collections [1]/[44.6s], total [44.5s]/[4.5h], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [63.7mb]-\>[65.9mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-06-04 14:50:14,937][INFO][cluster.service] [BUS2F2801F3] removed {[ELS-10.76.125.37][j3VQFYDaQLujkprUnke02w][inet[/10.76.125.37:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.38][5V8bqkEzTP2TzMukB5\_j-Q][inet[/10.76.122.38:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.48][TGlF1uv8Q5GpgBVvIcvRAQ][inet[/10.76.125.48:9300]]{max\_local\_storage\_nodes=1, master=false},[EDSFB1ABF7][MqLDnM5mSLqIicIuyJk7IQ][inet[/10.76.122.19:9300]]{client=true, data=false, master=false},[ELS-10.76.120.62][evcNI2CqSs-Zz44Jdzn0aw][inet[/10.76.120.62:9300]]{client=true, data=false, max\_local\_storage\_nodes=1, master=false},[BUS9364B62][YZPjEsvhT6OjM9ti5Lxwkg][inet[/10.76.123.123:9300]]{client=true, data=false, master=false},[ELS-10.76.125.38][RyeswSy8SquV5H8Vfsw75Q][inet[/10.76.125.38:9300]]{max\_local\_storage\_nodes=1, master=false},[EDSFB1200C][XUNaWVlYQUOVZlJMv3nHMA][inet[/10.76.122.18:9300]]{client=true, data=false, master=false},[ELS-10.76.124.214][H8N9nIU0TKyGv\_prKyRVCQ][inet[/10.76.124.214:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1F2240][ET2u1qImQCCvqc-1gRvQbQ][inet[/10.76.120.87:9300]]{client=true, data=false, master=false},[ELS-10.76.125.40][hp4wvQxER-mMPygey2Iqgg][inet[/10.76.125.40:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.67][BiXop5iCRgGQyGvxazMkQg][inet[/10.76.122.67:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.121.129][pf9xpva7Q4izIy6Nj4S4iQ][inet[/10.76.121.129:9300]]{data=false, max\_local\_storage\_nodes=1, master=true},[EDSFB21E69][RabnwdLbT1WCp9gIE-\_AXw][inet[/10.76.122.20:9300]]{client=true, data=false, master=false},[EDI1AE4FD76][UF1RMWe6RYaZGp6BU3x-VA][inet[/10.76.124.228:9300]]{client=true, data=false, master=false},[ELS-10.76.125.46][nXceQp40TjOSctChaGVtKw][inet[/10.76.125.46:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1A1EA928][rWlelgQuT7KHSfyIejmLPg][inet[/10.76.120.82:9300]]{client=true, data=false, master=false},[ELS-10.76.121.188][oWldDeY4TJioki90moNySw][inet[/10.76.121.188:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.34][kPSYm9G8R8i\_z2skK\_jq1g][inet[/10.76.122.34:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.43][JMgOIZFBSzaQZ9bVagG57w][inet[/10.76.125.43:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1AE3EE57][7JHGaYjzS3uI7PLN8Ynm-Q][inet[/10.76.124.227:9300]]{client=true, data=false, master=false},[ELS-10.76.124.225][nTPlE6IkTHOZ7EThX-hLeQ][inet[/10.76.124.225:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.120.61][\_60f636\_QsOPIWN0tKyN2A][inet[/10.76.120.61:9300]]{client=true, data=false, max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.47][MV8eSvpbRtCS1MAK2iAcVg][inet[/10.76.125.47:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1AB0123F][Di8rrVJMSYm6PVnAVuFnkw][inet[/10.76.124.18:9300]]{client=true, data=false, master=false},[BUS936E1B3][Vnr\_UCzOTtysBzM6NlhvFA][inet[/10.76.123.122:9300]]{client=true, data=false, master=false},[ELS-10.76.121.186][h8J-VleORCGTU8WbfnIuEw][inet[/10.76.121.186:9300]]{max\_local\_storage\_nodes=1, master=false},[EDI1A1EC098][SvH6eHsHRoyz8\_PEig46LA][inet[/10.76.120.83:9300]]{client=true, data=false, master=false},[ELS-10.76.121.131][dg\_r12\_nQbqIT\_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false, max\_local\_storage\_nodes=1, master=true},[ELS-10.76.122.39][swxUZAjCTCeBZS2-Uo\_Bpw][inet[/10.76.122.39:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.120.63][CsOrvy-3SoSewoSnj4Eyzw][inet[/10.76.120.63:9300]]{client=true, data=false, max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.45][LZtJHDJ3TQqTbCOnqyFAHw][inet[/10.76.125.45:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.37][TBw4vykLR\_uRT76gt23UUQ][inet[/10.76.122.37:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.36][uMjtmuRzQUW9i09Og\_DdvQ][inet[/10.76.122.36:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1F0AD0][gskAvZTNSaOVzanxgKi\_iA][inet[/10.76.120.86:9300]]{client=true, data=false, master=false},[ELS-10.76.125.50][jv7xpDnpTiOo21Kg\_JgkVw][inet[/10.76.125.50:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.124.223][WeuqeKYaS--2lGGpfgeV7w][inet[/10.76.124.223:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1EDBF0][iLruKxhKQymkiaAKz0GXaw][inet[/10.76.120.84:9300]]{client=true, data=false, master=false},[ELS-10.76.124.221][nB65C4WGRiSCntVZoovX\_Q][inet[/10.76.124.221:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.124.224][fxfz13VnSPOs4lv6BGelPQ][inet[/10.76.124.224:9300]]{max\_local\_storage\_nodes=1, master=false},[EDSFB2D1AD][TFqeE7LWTvqKOrhSPg3wcg][inet[/10.76.122.21:9300]]{client=true, data=false, master=false},[STG0A4C78E2][ls59ftO9TyiKVZvuc6jeRg][inet[/10.76.120.226:9300]]{client=true, data=false, master=false},[ELS-10.76.124.218][0hShFOjuSvCEmR1LW6jQSQ][inet[/10.76.124.218:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.125.36][ioXkziSRTRKeml-4oXRNRw][inet[/10.76.125.36:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1DFC2FCE][5Js9J7v7RhCNJx2AhMaQKw][inet[/10.76.121.248:9300]]{client=true, data=false, master=false},[ELS-10.76.125.49][IeM2iMC4SbuzPWcsgTrR1w][inet[/10.76.125.49:9300]]{max\_local\_storage\_nodes=1, master=false},[ELS-10.76.122.35][usJjEG65Tk6D4YnVHz220w][inet[/10.76.122.35:9300]]{max\_local\_storage\_nodes=1, master=false},[EDS1A1EF360][eucRgy56R1uf2BnV6GILZQ][inet[/10.76.120.85:9300]]{client=true, data=false, master=false},[EDSF899A97][Mg-eNFm7TuSUutzerCtcLQ][inet[/10.76.122.17:9300]]{client=true, data=false, master=false},}, reason: zen-disco-master\_failed ([ELS-10.76.121.131][dg\_r12\_nQbqIT\_oJfjTwTg][inet[/10.76.121.131:9300]]{data=false, max\_local\_storage\_nodes=1, master=true}) [2014-06-04 14:52:25,048][WARN][monitor.jvm] [BUS2F2801F3] [gc][old][55507][493] duration [43.7s], collections [1]/[43.7s], total [43.7s]/[4.5h], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [65.9mb]-\>[66.3mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} [2014-06-04 15:07:08,535][WARN ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2014-06-04 15:07:08,535][DEBUG][action.search.type] [BUS2F2801F3] [1004\_exposureindex][49], node[ot8ch3V6TYGKOLYnCY9cYw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@14495084] lastShard [true] org.elasticsearch.transport.SendRequestTransportException: [ELS-10.76.125.44][inet[/10.76.125.44:9300]][search/phase/query] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:202) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:173) at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:208) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.transport.NodeNotConnectedException: [ELS-10.76.125.44][inet[/10.76.125.44:9300]] Node not connected at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:859) at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:540) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:189) ... 8 more On Tuesday, 27 May 2014 14:03:29 UTC-7, VB wrote: > Hi all, > > We are running 90.11 and we have a cluster with client, master and data > nodes. > > Our client nodes are using dedicated 10g memory. > > But we are seeing these outofmemory exceptions. > > I tried to correlate this log time with logs in our exception but I did > not find any query which we could be causing this issue. > > Our cluster has 37 indexes with 50 shards and 1 replica. > > Some indexes has data and some doesn't. > > [2014-05-27 16:26:34,688][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327409][8] duration [33s], collections [1]/[33.5s], total > [33s]/[3.8m], memory [9gb]-\>[9.4gb]/[9.9gb], all\_pools {[young] > [520.2kb]-\>[112.3mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:27:19,992][INFO][cluster.service] [BUS9364B62] > detected\_master > [ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}, added > {[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true},}, reason: zen-disco-receive(from > master > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}]) > [2014-05-27 16:27:20,008][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:27:20,008][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327410][9] duration [44.3s], collections [1]/[45.3s], total > [44.3s]/[4.6m], memory [9.4gb]-\>[9.8gb]/[9.9gb], all\_pools {[young] > [112.3mb]-\>[475mb]/[532.5mb]}{[survivor] [0b]-\>[0b]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:06,856][WARN][cluster.service] [BUS9364B62] > failed to connect to node > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}] > org.elasticsearch.transport.ConnectTransportException: > [ELS-10.76.121.130][inet[/10.76.121.130:9300]] connect\_timeout[30s] > at > org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:727) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:647) > at > org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:615) > at > org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129) > at > org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:396) > at > org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:135) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > [2014-05-27 16:28:06,856][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327412][10] duration [45.3s], collections [1]/[45.8s], total > [45.3s]/[5.3m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] [41mb]-\>[33.2mb]/[66.5mb]}{[old] > [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:28:53,876][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327413][11] duration [46.4s], collections [1]/[47s], total > [46.4s]/[6.1m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [33.2mb]-\>[54.7mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:29:40,990][INFO][discovery.zen] [BUS9364B62] > master\_left > [[ELS-10.76.121.130][BlGygpFmRn6uQNbgiEfl0A][inet[/10.76.121.130:9300]]{data=false, > max\_local\_storage\_nodes=1, master=true}], reason [failed to perform initial > connect [[ELS-10.76.121.130][inet[/10.76.121.130:9300]] > connect\_timeout[30s]]] > [2014-05-27 16:29:40,990][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327414][12] duration [46.8s], collections [1]/[47.1s], total > [46.8s]/[6.9m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [54.7mb]-\>[65.5mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:30:27,589][WARN][monitor.jvm] [BUS9364B62] > [gc][old][327415][13] duration [46.5s], collections [1]/[46.5s], total > [46.5s]/[7.7m], memory [9.9gb]-\>[9.9gb]/[9.9gb], all\_pools {[young] > [532.5mb]-\>[532.5mb]/[532.5mb]}{[survivor] > [65.5mb]-\>[66.4mb]/[66.5mb]}{[old] [9.3gb]-\>[9.3gb]/[9.3gb]} > [2014-05-27 16:48:27,804][WARN > ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the > selector loop. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.iterator(ArrayList.java:814) > at > sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:496) > at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:172) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at > org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a73f803b-1009-4bb4-9ce8-1693a646a82e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a73f803b-1009-4bb4-9ce8-1693a646a82e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:24am UTC](https://discuss.elastic.co/t/outofmemory-exception-on-client-node/17764/11 "2017-07-06T01:24:42Z")