# Node not available exception is occurred after Out of memory error
**URL:** https://discuss.elastic.co/t/node-not-available-exception-is-occurred-after-out-of-memory-error/11070
**Category:** Elasticsearch
**Created:** [March 8, 2013, 7:25am UTC](https://discuss.elastic.co/t/node-not-available-exception-is-occurred-after-out-of-memory-error/11070 "2013-03-08T07:25:15Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@hoong](https://discuss.elastic.co/u/hoong)
#### Post date: [March 8, 2013, 7:25am UTC](https://discuss.elastic.co/t/node-not-available-exception-is-occurred-after-out-of-memory-error/11070/1 "2013-03-08T07:25:15Z")
Hello.
I run ES on 4 nodes with 4 shards.
Each node are run on the 8 core cpu with 16 GB memory, and I set mem heap
size 8GB for ES node. OS is Linux CentOS 5.3.
I indexed 350 GB documents, and its counts almost 500 milions.
Final index size of all four shards reach 380 GB.
## The node settings like follows,...
index.number\_of\_shards: 4
index.number\_of\_replicas: 0
index.cache.field.type: soft
bootstrap.mlockall: true
indices.memory.index\_buffer\_size: 10% \<- when I first create cluster, it
starts with 70% to enhance indexing speed, and now I changed the value from
70% to 10% and just restart all nodes. Is it enough to change setting?
gateway.recover\_after\_nodes: 3
gateway.recover\_after\_time: 3m
gateway.expected\_nodes: 4
## discovery.zen.ping.timeout: 3s discovery.zen.ping.unicast.hosts: 1.234.83.104:9300, 211.110.1.24:9300, 211.110.1.20:9300, 1.234.83.149:9300
My question is,
When we requests facet search on the node with field[d\_ip], than OOM is
occured on node, and then cluster crashes until I restart nodes manually.
There are so many kinds of d\_ip would exist i know. However,..
_Is it normal that nodes are all going down when OOM is occurred by Query?_
_Is it normal that nodes are remained crashing when OOM is occurred on
nodes?_
I got other many performance issues, but this stability issue is mostly
important for me.
please give me answers..
regards.
_Below logs are our debug log file._
org.elasticsearch.transport.RemoteTransportException:
[bingo\_node2][inet[/211.110.1.24:9300]][search/phase/query]
Caused by: java.lang.OutOfMemoryError: loading field [d\_ip] caused out of
memory failure
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:138)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:81)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:178)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:242)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:529)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:518)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2219)
at java.util.ArrayList.grow(ArrayList.java:213)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:187)
at java.util.ArrayList.add(ArrayList.java:411)
at
org.elasticsearch.index.field.data.strings.StringFieldData$StringTypeLoader.collectTerm(StringFieldData.java:105)
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:59)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:130)
... 17 more
[2013-03-08 13:47:21,539][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][333][46] duration [50.6s], collections
[3]/[50.6s], total [50.6s]/[12.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.8mb]-\>[29.8mb]/[82mb]}
[2013-03-08 13:47:52,520][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [50560ms] ago,
timed out [215ms] ago, action [discovery/zen/fd/masterPing], node
[[bingo\_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true}],
id [416]
[2013-03-08 13:48:11,419][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][334][49] duration [49.8s], collections
[3]/[49.8s], total [49.8s]/[13.8m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.8mb]-\>[29.8mb]/[82mb]}
[2013-03-08 13:48:20,824][INFO][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][335][50] duration [9.4s], collections [1]/[9.4s],
total [9.4s]/[13.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.8mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:48:36,713][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][336][51] duration [15.8s], collections
[1]/[15.8s], total [15.8s]/[14.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:49:02,503][INFO][discovery.zen] [bingo\_node0]
master\_left
[[bingo\_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true}],
reason [do not exists on master, act as master failure]
[2013-03-08 13:49:02,505][INFO][cluster.service] [bingo\_node0]
master {new
[bingo\_node0][zosv96W9QKmAZ0J7PulqQw][inet[/1.234.83.149:9300]]{master=true},
previous
[bingo\_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true}},
removed
{[bingo\_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true},},
reason: zen-disco-master\_failed
([bingo\_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true})
[2013-03-08 13:49:02,505][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][337][53] duration [25.7s], collections
[2]/[25.7s], total [25.7s]/[14.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]-\>[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:49:45,263][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][338][55] duration [30.8s], collections
[2]/[30.8s], total [30.8s]/[15.1m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:50:22,982][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][339][59] duration [49.6s], collections
[4]/[49.6s], total [49.6s]/[16m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[65.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:50:42,287][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][340][60] duration [19.2s], collections
[1]/[19.3s], total [19.2s]/[16.3m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.6mb]-\>[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:50:54,270][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][341][61] duration [11.9s], collections
[1]/[11.9s], total [11.9s]/[16.5m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]-\>[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[29.9mb]/[82mb]}
[2013-03-08 13:50:54,271][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [80919ms] ago,
timed out [31287ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node1][dw57v59OQeGn\_LuII9z7hg][inet[1.234.83.104/1.234.83.104:9300]]{master=false}],
id [421]
[2013-03-08 13:51:13,017][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][342][62] duration [18.7s], collections
[1]/[18.7s], total [18.7s]/[16.8m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]-\>[30mb]/[82mb]}
[2013-03-08 13:51:25,000][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][343][63] duration [11.9s], collections
[1]/[11.9s], total [11.9s]/[17m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:51:53,522][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][344][65] duration [28.3s], collections
[2]/[28.3s], total [28.3s]/[17.5m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:52:49,899][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][345][68] duration [37.8s], collections
[3]/[37.8s], total [37.8s]/[18.1m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:54:01,672][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][346][73] duration [1m], collections [5]/[1m],
total [1m]/[19.1m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:54:17,875][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][347][77] duration [44s], collections [4]/[44s],
total [44s]/[19.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:55:04,347][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [349785ms] ago,
timed out [319083ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][4guTyorqTUe0mEtoWocYQw][inet[211.110.1.20/211.110.1.20:9300]]{master=false}],
id [420]
[2013-03-08 13:55:22,965][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][348][80] duration [34.4s], collections
[3]/[34.4s], total [34.4s]/[20.4m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:55:22,965][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [337701ms] ago,
timed out [299982ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][4guTyorqTUe0mEtoWocYQw][inet[211.110.1.20/211.110.1.20:9300]]{master=false}],
id [422]
[2013-03-08 13:55:32,168][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][349][83] duration [39.8s], collections
[3]/[39.8s], total [39.8s]/[21.1m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:56:00,358][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [327973ms] ago,
timed out [296685ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][4guTyorqTUe0mEtoWocYQw][inet[211.110.1.20/211.110.1.20:9300]]{master=false}],
id [423]
[2013-03-08 13:57:11,212][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [152115ms] ago,
timed out [0ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node1][dw57v59OQeGn\_LuII9z7hg][inet[1.234.83.104/1.234.83.104:9300]]{master=false}],
id [429]
[2013-03-08 13:57:29,750][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][350][90] duration [1.6m], collections [7]/[1.6m],
total [1.6m]/[22.8m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 13:57:39,153][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][351][92] duration [27.9s], collections
[2]/[27.9s], total [27.9s]/[23.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [3.1mb]-\>[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]-\>[30mb]/[82mb]}
[2013-03-08 14:01:35,807][WARN][transport.netty] [bingo\_node0]
exception caught on transport layer [[id: 0x04281a69, /211.110.1.20:40478
=\> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.compress.BufferRecycler.allocEncodingHash(BufferRecycler.java:95)
at
org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java:72)
at
org.elasticsearch.common.compress.lzf.LZFCompressedStreamOutput.(LZFCompressedStreamOutput.java:42)
at
org.elasticsearch.common.compress.lzf.LZFCompressor.streamOutput(LZFCompressor.java:133)
at
org.elasticsearch.common.io.stream.CachedStreamOutput$Entry.handles(CachedStreamOutput.java:72)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:83)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:67)
at
org.elasticsearch.cluster.action.index.NodeAliasesUpdatedAction$NodeAliasesUpdatedTransportHandler.messageReceived(NodeAliasesUpdatedAction.java:117)
at
org.elasticsearch.cluster.action.index.NodeAliasesUpdatedAction$NodeAliasesUpdatedTransportHandler.messageReceived(NodeAliasesUpdatedAction.java:105)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[2013-03-08 14:05:37,845][WARN][transport.netty] [bingo\_node0]
exception caught on transport layer [[id: 0xa7dd9a40, /211.110.1.20:40500
=\> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 14:05:19,247][WARN][transport.netty] [bingo\_node0]
exception caught on transport layer [[id: 0xc2f655a6, /211.110.1.20:40498
=\> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 14:29:23,624][INFO][node] [bingo\_node0]
{0.20.5}[19651]: initializing ...
[2013-03-08 14:29:23,642][INFO][plugins] [bingo\_node0]
loaded [], sites [head]
[2013-03-08 14:29:26,042][INFO][node] [bingo\_node0]
{0.20.5}[19651]: initialized
[2013-03-08 14:29:26,043][INFO][node] [bingo\_node0]
{0.20.5}[19651]: starting ...
[2013-03-08 14:29:26,165][INFO][transport] [bingo\_node0]
bound\_address {inet[/1.234.83.149:9300]}, publish\_address
{inet[/1.234.83.149:9300]}
[2013-03-08 14:29:29,272][INFO][cluster.service] [bingo\_node0]
new\_master
[bingo\_node0][auZIU6m1Rb2Cf0GOe\_YYhQ][inet[/1.234.83.149:9300]]{master=true},
reason: zen-disco-join (elected\_as\_master)
[2013-03-08 14:29:29,279][INFO][discovery] [bingo\_node0]
bingo\_dist/auZIU6m1Rb2Cf0GOe\_YYhQ
[2013-03-08 14:29:29,294][INFO][http] [bingo\_node0]
bound\_address {inet[/1.234.83.149:9200]}, publish\_address
{inet[/1.234.83.149:9200]}
[2013-03-08 14:29:29,294][INFO][node] [bingo\_node0]
{0.20.5}[19651]: started
[2013-03-08 14:29:51,057][INFO][cluster.service] [bingo\_node0]
added
{[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false},},
reason: zen-disco-receive(join from
node[[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}])
[2013-03-08 14:30:11,625][INFO][cluster.service] [bingo\_node0]
added
{[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false},},
reason: zen-disco-receive(join from
node[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}])
[2013-03-08 14:30:28,377][INFO][cluster.service] [bingo\_node0]
added
{[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true},},
reason: zen-disco-receive(join from
node[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}])
[2013-03-08 14:30:28,895][INFO][gateway] [bingo\_node0]
recovered [1] indices into cluster\_state
[2013-03-08 14:41:56,724][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][738][1] duration [10.1s], collections
[1]/[11.1s], total [10.1s]/[10.2s], memory [7.1gb]-\>[6.6gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[102.9mb]-\>[20.1mb]/[532.5mb]}{[Par Survivor Space]
[66.5mb]-\>[0b]/[66.5mb]}{[CMS Old Gen] [6.9gb]-\>[6.6gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:42:13,375][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][741][2] duration [13.9s], collections
[1]/[14.6s], total [13.9s]/[24.1s], memory [7.1gb]-\>[7.1gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[252.8mb]-\>[11.5mb]/[532.5mb]}{[Par Survivor Space]
[66.5mb]-\>[0b]/[66.5mb]}{[CMS Old Gen] [6.8gb]-\>[7gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:42:31,243][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][743][3] duration [16.7s], collections
[1]/[16.8s], total [16.7s]/[40.9s], memory [7.5gb]-\>[7.3gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[507.6mb]-\>[232.9mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:42:50,510][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][744][4] duration [18.4s], collections
[1]/[19.2s], total [18.4s]/[59.3s], memory [7.3gb]-\>[7.5gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[232.9mb]-\>[391.9mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:43:09,926][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][745][5] duration [18.6s], collections
[1]/[19.4s], total [18.6s]/[1.2m], memory [7.5gb]-\>[7.6gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[391.9mb]-\>[500mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:43:29,541][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][746][6] duration [19s], collections [1]/[19.6s],
total [19s]/[1.6m], memory [7.6gb]-\>[7.6gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[500mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[10.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:43:49,239][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][747][7] duration [19.2s], collections
[1]/[19.6s], total [19.2s]/[1.9m], memory [7.6gb]-\>[7.2gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[80.8mb]/[532.5mb]}{[Par Survivor Space]
[10.8mb]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:44:09,094][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][749][8] duration [18.3s], collections
[1]/[18.7s], total [18.3s]/[2.2m], memory [7.6gb]-\>[7.4gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[338.9mb]/[532.5mb]}{[Par Survivor Space]
[7.9mb]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:44:26,939][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][750][9] duration [16.9s], collections
[1]/[17.9s], total [16.9s]/[2.5m], memory [7.4gb]-\>[7.6gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[338.9mb]-\>[472.9mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:44:46,945][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][751][10] duration [19.3s], collections [1]/[20s],
total [19.3s]/[2.8m], memory [7.6gb]-\>[7.6gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[472.9mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[272kb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:45:06,752][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][752][11] duration [19.1s], collections
[1]/[19.7s], total [19.1s]/[3.1m], memory [7.6gb]-\>[7.4gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[244.9mb]/[532.5mb]}{[Par Survivor Space]
[272kb]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:45:27,857][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][753][12] duration [20.3s], collections
[1]/[21.2s], total [20.3s]/[3.5m], memory [7.4gb]-\>[7.5gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[244.9mb]-\>[416.2mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:45:49,213][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][754][13] duration [20.7s], collections
[1]/[21.3s], total [20.7s]/[3.8m], memory [7.5gb]-\>[7.6gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[416.2mb]-\>[496.8mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:46:10,818][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][755][14] duration [21s], collections [1]/[21.6s],
total [21s]/[4.2m], memory [7.6gb]-\>[7.6gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[496.8mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[0b]-\>[13.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:46:32,436][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][756][15] duration [21.2s], collections
[1]/[21.6s], total [21.2s]/[4.5m], memory [7.6gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[13.6mb]-\>[21.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:46:53,478][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][757][16] duration [20.7s], collections [1]/[21s],
total [20.7s]/[4.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[21.8mb]-\>[42.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:47:12,208][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][758][17] duration [18.5s], collections
[1]/[18.7s], total [18.5s]/[5.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[42.8mb]-\>[54.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:47:33,478][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][759][18] duration [21.1s], collections
[1]/[21.2s], total [21.1s]/[5.5m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[54.8mb]-\>[60.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:47:55,086][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][760][19] duration [21.5s], collections
[1]/[21.6s], total [21.5s]/[5.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[60.6mb]-\>[63mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:48:16,660][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][761][20] duration [21.5s], collections
[1]/[21.5s], total [21.5s]/[6.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[63mb]-\>[64.5mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:48:36,173][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][762][21] duration [19.5s], collections
[1]/[19.5s], total [19.5s]/[6.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[64.5mb]-\>[65.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:48:55,328][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][763][22] duration [19.1s], collections
[1]/[19.1s], total [19.1s]/[6.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.1mb]-\>[65.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:49:14,624][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][764][23] duration [19.2s], collections
[1]/[19.2s], total [19.2s]/[7.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.6mb]-\>[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:49:14,625][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [57965ms] ago,
timed out [19296ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2203]
[2013-03-08 14:49:34,278][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][765][24] duration [19.4s], collections
[1]/[19.4s], total [19.4s]/[7.5m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]-\>[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:49:56,381][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][766][25] duration [22s], collections [1]/[22.1s],
total [22s]/[7.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]-\>[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:50:18,383][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][767][26] duration [22.2s], collections
[1]/[22.2s], total [22.2s]/[8.3m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]-\>[65.5mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:50:18,599][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [44321ms] ago,
timed out [215ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2208]
[2013-03-08 14:50:18,599][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [63975ms] ago,
timed out [22218ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2207]
[2013-03-08 14:50:40,412][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][768][27] duration [21.8s], collections
[1]/[21.8s], total [21.8s]/[8.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.5mb]-\>[65mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:51:02,454][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][769][28] duration [22.2s], collections
[1]/[22.2s], total [22.2s]/[9m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools
{[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65mb]-\>[65.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:51:02,454][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [66073ms] ago,
timed out [22041ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2210]
[2013-03-08 14:51:24,756][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][770][29] duration [22.2s], collections
[1]/[22.3s], total [22.2s]/[9.4m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.4mb]-\>[65.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:51:24,971][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [44559ms] ago,
timed out [215ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2212]
[2013-03-08 14:51:47,120][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][771][30] duration [22.3s], collections
[1]/[22.3s], total [22.3s]/[9.7m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.8mb]-\>[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:52:06,644][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [64191ms] ago,
timed out [19523ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2214]
[2013-03-08 14:52:59,425][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][772][32] duration [31.3s], collections
[2]/[31.3s], total [31.3s]/[10.3m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:53:20,698][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][773][35] duration [1m], collections [3]/[1m],
total [1m]/[11.3m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:53:39,328][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][774][36] duration [18.8s], collections
[1]/[18.8s], total [18.8s]/[11.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:54:13,136][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [73498ms] ago,
timed out [33808ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2218]
[2013-03-08 14:54:22,528][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][775][38] duration [33.8s], collections
[2]/[33.8s], total [33.8s]/[12.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:55:15,161][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [95834ms] ago,
timed out [21282ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2221]
[2013-03-08 14:55:15,161][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [95833ms] ago,
timed out [52633ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2222]
[2013-03-08 14:55:49,218][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][776][43] duration [1.4m], collections [5]/[1.4m],
total [1.4m]/[13.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:56:10,588][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][777][45] duration [33.2s], collections
[2]/[33.2s], total [33.2s]/[14.1m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:56:10,589][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [55214ms] ago,
timed out [21370ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2225]
[2013-03-08 14:56:44,149][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [100774ms] ago,
timed out [33367ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2223]
[2013-03-08 14:57:17,844][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][778][48] duration [45.5s], collections
[3]/[45.5s], total [45.5s]/[14.9m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:58:08,143][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [138924ms] ago,
timed out [50299ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2226]
[2013-03-08 14:58:17,395][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [114810ms] ago,
timed out [9252ms] ago, action [/gateway/local/started-shards/n], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2228]
[2013-03-08 14:58:17,395][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][779][53] duration [1.3m], collections [5]/[1.3m],
total [1.3m]/[16.2m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]-\>[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:58:38,929][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][780][54] duration [21.7s], collections
[1]/[21.7s], total [21.7s]/[16.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]-\>[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29mb]/[82mb]}
[2013-03-08 14:58:38,929][INFO][cluster.service] [bingo\_node0]
removed
{[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false},},
reason:
zen-disco-node\_failed([bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}),
reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-03-08 14:59:53,034][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [188885ms] ago,
timed out [95636ms] ago, action [/gateway/local/started-shards/n], node
[[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2229]
[2013-03-08 14:59:53,034][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][781][58] duration [52.3s], collections
[4]/[52.3s], total [52.3s]/[17.5m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]-\>[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]-\>[29.1mb]/[82mb]}
[2013-03-08 15:00:05,033][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][782][60] duration [33.7s], collections
[2]/[33.7s], total [33.7s]/[18m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]-\>[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.1mb]-\>[29.1mb]/[82mb]}
[2013-03-08 15:00:26,606][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][783][61] duration [21.5s], collections
[1]/[21.5s], total [21.5s]/[18.4m], memory [7.7gb]-\>[7.7gb]/[7.7gb],
all\_pools {[Code Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]-\>[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.1mb]-\>[29.1mb]/[82mb]}
[2013-03-08 15:00:38,655][DEBUG][action.search.type] [bingo\_node0]
[20130305][1], node[nXyQPEDuT4qXlUlXcW2ALQ], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@63012d7c]
org.elasticsearch.transport.NodeDisconnectedException:
[bingo\_node3][inet[/211.110.1.20:9300]][search/phase/query] disconnected
[2013-03-08 15:01:00,382][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [150513ms] ago,
timed out [119727ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2231]
[2013-03-08 15:01:00,382][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][784][62] duration [12s], collections [1]/[12s],
total [12s]/[18.6m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]-\>[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.1mb]-\>[29.2mb]/[82mb]}
[2013-03-08 15:01:33,692][WARN][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][785][65] duration [55s], collections [3]/[55s],
total [55s]/[19.5m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]-\>[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.2mb]-\>[29.2mb]/[82mb]}
[2013-03-08 15:01:43,058][INFO][monitor.jvm] [bingo\_node0]
[gc][ConcurrentMarkSweep][786][66] duration [9.3s], collections [1]/[9.3s],
total [9.3s]/[19.7m], memory [7.7gb]-\>[7.7gb]/[7.7gb], all\_pools {[Code
Cache] [1.3mb]-\>[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]-\>[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]-\>[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]-\>[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.2mb]-\>[29.2mb]/[82mb]}
[2013-03-08 15:02:04,355][INFO][cluster.service] [bingo\_node0]
removed
{[bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false},},
reason:
zen-disco-node\_failed([bingo\_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}),
reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-03-08 15:02:04,356][DEBUG][action.search.type] [bingo\_node0]
[20130305][3], node[5OkiRwVySNukFAxK8J68cg], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@63012d7c]
org.elasticsearch.transport.NodeDisconnectedException:
[bingo\_node1][inet[/1.234.83.104:9300]][search/phase/query] disconnected
[2013-03-08 15:02:49,890][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [250747ms] ago,
timed out [176641ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2232]
[2013-03-08 15:06:10,473][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [310092ms] ago,
timed out [276781ms] ago, action [/gateway/local/started-shards/n], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2237]
[2013-03-08 15:08:05,076][WARN][transport] [bingo\_node0]
Received response for a request that has timed out, sent [492041ms] ago,
timed out [446421ms] ago, action [discovery/zen/fd/ping], node
[[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2236]
[2013-03-08 15:18:02,663][WARN][transport.netty] [bingo\_node0]
exception caught on transport layer [[id: 0x4f47082e, /211.110.1.24:35454
=\> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.compress.BufferRecycler.allocEncodingHash(BufferRecycler.java:95)
at
org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java:72)
at
org.elasticsearch.common.compress.lzf.LZFCompressedStreamOutput.(LZFCompressedStreamOutput.java:42)
at
org.elasticsearch.common.compress.lzf.LZFCompressor.streamOutput(LZFCompressor.java:133)
at
org.elasticsearch.common.io.stream.CachedStreamOutput$Entry.handles(CachedStreamOutput.java:72)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:83)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:67)
at
org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler.messageReceived(MasterFaultDetection.java:387)
at
org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler.messageReceived(MasterFaultDetection.java:362)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[2013-03-08 15:18:48,782][INFO][cluster.service] [bingo\_node0]
removed
{[bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true},},
reason:
zen-disco-node\_failed([bingo\_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}),
reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-03-08 15:18:36,721][DEBUG][action.search.type] [bingo\_node0]
[20130305][0], node[hXCZyzPzRRiNjl1KdIRegA], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@63012d7c]
org.elasticsearch.transport.NodeDisconnectedException:
[bingo\_node2][inet[/211.110.1.24:9300]][search/phase/query] disconnected
[2013-03-08 15:31:43,246][WARN][index.engine.robin] [bingo\_node0]
[20130305][2] failed engine
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 15:32:11,408][WARN][netty.channel.DefaultChannelPipeline] An
exception was thrown by a user handler while handling an exception event
([id: 0xc26ff33f, /1.234.83.149:45454 =\> /1.234.83.149:9300] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 15:37:53,195][INFO][node] [bingo\_node0]
{0.20.5}[20685]: initializing ...
[2013-03-08 15:37:53,199][INFO][plugins] [bingo\_node0]
loaded [], sites [head]
[2013-03-08 15:37:55,111][INFO][node] [bingo\_node0]
{0.20.5}[20685]: initialized
[2013-03-08 15:37:55,111][INFO][node] [bingo\_node0]
{0.20.5}[20685]: starting ...
[2013-03-08 15:37:55,207][INFO][transport] [bingo\_node0]
bound\_address {inet[/1.234.83.149:9300]}, publish\_address
{inet[/1.234.83.149:9300]}
[2013-03-08 15:37:59,062][WARN][discovery.zen.ping.unicast] [bingo\_node0]
failed to send ping to [[#zen\_unicast\_3#][inet[/211.110.1.24:9300]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[/211.110.1.24:9300]][discovery/zen/unicast] request\_id [1] timed
out after [3752ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2013-03-08 15:37:59,811][INFO][cluster.service] [bingo\_node0]
new\_master
[bingo\_node0][FgOry\_9FTsynZMJXZVnkwA][inet[/1.234.83.149:9300]]{master=true},
reason: zen-disco-join (elected\_as\_master)
[2013-03-08 15:37:59,818][INFO][discovery] [bingo\_node0]
bingo\_dist/FgOry\_9FTsynZMJXZVnkwA
[2013-03-08 15:37:59,834][INFO][http] [bingo\_node0]
bound\_address {inet[/1.234.83.149:9200]}, publish\_address
{inet[/1.234.83.149:9200]}
[2013-03-08 15:37:59,834][INFO][node] [bingo\_node0]
{0.20.5}[20685]: started
[2013-03-08 15:38:03,840][INFO][cluster.service] [bingo\_node0]
added
{[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false},},
reason: zen-disco-receive(join from
node[[bingo\_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}])
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [March 8, 2013, 11:19am UTC](https://discuss.elastic.co/t/node-not-available-exception-is-occurred-after-out-of-memory-error/11070/2 "2013-03-08T11:19:00Z")
My observations have been almost the same, unfortunately. In such
situations like OOM, internal state of a node might get corrupted sooner
or later, and a node must be restarted to recover as soon as possible.
I admit I want to start a plugin "elasticsearch-low-memory-detector"
that can detect low resource situations in advance. The idea is to
completely avoid OOMs (or other dangerous situations) by giving warnings
in advance to the admin. There should be a reasonable set of default
thresholds to check the various JVM heap info and RuntimeMXBean values,
of course configurable. Also actions should be configured for adding
resource warning texts to the node attributes for evaluation by external
monitoring tools, sending out emails to the admin, emergency stopping of
the node, etc. Blocking "bad queries" like facet requests on high
cardinality fields would also be nice - but I have no good idea how to
do precompute field cache allocation now...
Jörg
Am 08.03.13 08:25, schrieb hoong:
> _Is it normal that nodes are all going down when OOM is occurred by
> Query?_
> _Is it normal that nodes are remained crashing when OOM is occurred on
> nodes?_
>
> I got other many performance issues, but this stability issue is
> mostly important for me.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/node-not-available-exception-is-occurred-after-out-of-memory-error/11070/3 "2017-07-06T02:47:36Z")