Hello.
I run ES on 4 nodes with 4 shards.
Each node are run on the 8 core cpu with 16 GB memory, and I set mem heap
size 8GB for ES node. OS is Linux CentOS 5.3.
I indexed 350 GB documents, and its counts almost 500 milions.
Final index size of all four shards reach 380 GB.
The node settings like follows,...
index.number_of_shards: 4
index.number_of_replicas: 0
index.cache.field.type: soft
bootstrap.mlockall: true
indices.memory.index_buffer_size: 10% <- when I first create cluster, it
starts with 70% to enhance indexing speed, and now I changed the value from
70% to 10% and just restart all nodes. Is it enough to change setting?
gateway.recover_after_nodes: 3
gateway.recover_after_time: 3m
gateway.expected_nodes: 4
discovery.zen.ping.timeout: 3s
discovery.zen.ping.unicast.hosts: 1.234.83.104:9300, 211.110.1.24:9300,
211.110.1.20:9300, 1.234.83.149:9300
My question is,
When we requests facet search on the node with field[d_ip], than OOM is
occured on node, and then cluster crashes until I restart nodes manually.
There are so many kinds of d_ip would exist i know. However,..
Is it normal that nodes are all going down when OOM is occurred by Query?
Is it normal that nodes are remained crashing when OOM is occurred on
nodes?
I got other many performance issues, but this stability issue is mostly
important for me.
please give me answers..
regards.
Below logs are our debug log file.
org.elasticsearch.transport.RemoteTransportException:
[bingo_node2][inet[/211.110.1.24:9300]][search/phase/query]
Caused by: java.lang.OutOfMemoryError: loading field [d_ip] caused out of
memory failure
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:138)
at
org.elasticsearch.search.facet.terms.strings.TermsStringOrdinalsFacetCollector.doSetNextReader(TermsStringOrdinalsFacetCollector.java:128)
at
org.elasticsearch.search.facet.AbstractFacetCollector.setNextReader(AbstractFacetCollector.java:81)
at
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:67)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:576)
at
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:195)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:426)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:342)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:330)
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:178)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:242)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:529)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:518)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2219)
at java.util.ArrayList.grow(ArrayList.java:213)
at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:187)
at java.util.ArrayList.add(ArrayList.java:411)
at
org.elasticsearch.index.field.data.strings.StringFieldData$StringTypeLoader.collectTerm(StringFieldData.java:105)
at
org.elasticsearch.index.field.data.support.FieldDataLoader.load(FieldDataLoader.java:59)
at
org.elasticsearch.index.field.data.strings.StringFieldData.load(StringFieldData.java:90)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:56)
at
org.elasticsearch.index.field.data.strings.StringFieldDataType.load(StringFieldDataType.java:34)
at org.elasticsearch.index.field.data.FieldData.load(FieldData.java:111)
at
org.elasticsearch.index.cache.field.data.support.AbstractConcurrentMapFieldDataCache.cache(AbstractConcurrentMapFieldDataCache.java:130)
... 17 more
[2013-03-08 13:47:21,539][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][333][46] duration [50.6s], collections
[3]/[50.6s], total [50.6s]/[12.9m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.8mb]->[29.8mb]/[82mb]}
[2013-03-08 13:47:52,520][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [50560ms] ago,
timed out [215ms] ago, action [discovery/zen/fd/masterPing], node
[[bingo_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true}],
id [416]
[2013-03-08 13:48:11,419][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][334][49] duration [49.8s], collections
[3]/[49.8s], total [49.8s]/[13.8m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.8mb]->[29.8mb]/[82mb]}
[2013-03-08 13:48:20,824][INFO ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][335][50] duration [9.4s], collections [1]/[9.4s],
total [9.4s]/[13.9m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.8mb]->[29.9mb]/[82mb]}
[2013-03-08 13:48:36,713][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][336][51] duration [15.8s], collections
[1]/[15.8s], total [15.8s]/[14.2m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[29.9mb]/[82mb]}
[2013-03-08 13:49:02,503][INFO ][discovery.zen ] [bingo_node0]
master_left
[[bingo_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true}],
reason [do not exists on master, act as master failure]
[2013-03-08 13:49:02,505][INFO ][cluster.service ] [bingo_node0]
master {new
[bingo_node0][zosv96W9QKmAZ0J7PulqQw][inet[/1.234.83.149:9300]]{master=true},
previous
[bingo_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true}},
removed
{[bingo_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true},},
reason: zen-disco-master_failed
([bingo_node2][Ez26deSDQB2S3KuNgzqiIw][inet[/211.110.1.24:9300]]{master=true})
[2013-03-08 13:49:02,505][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][337][53] duration [25.7s], collections
[2]/[25.7s], total [25.7s]/[14.6m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]->[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[29.9mb]/[82mb]}
[2013-03-08 13:49:45,263][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][338][55] duration [30.8s], collections
[2]/[30.8s], total [30.8s]/[15.1m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[29.9mb]/[82mb]}
[2013-03-08 13:50:22,982][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][339][59] duration [49.6s], collections
[4]/[49.6s], total [49.6s]/[16m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[65.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[29.9mb]/[82mb]}
[2013-03-08 13:50:42,287][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][340][60] duration [19.2s], collections
[1]/[19.3s], total [19.2s]/[16.3m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.6mb]->[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[29.9mb]/[82mb]}
[2013-03-08 13:50:54,270][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][341][61] duration [11.9s], collections
[1]/[11.9s], total [11.9s]/[16.5m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]->[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[29.9mb]/[82mb]}
[2013-03-08 13:50:54,271][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [80919ms] ago,
timed out [31287ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node1][dw57v59OQeGn_LuII9z7hg][inet[1.234.83.104/1.234.83.104:9300]]{master=false}],
id [421]
[2013-03-08 13:51:13,017][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][342][62] duration [18.7s], collections
[1]/[18.7s], total [18.7s]/[16.8m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.9mb]->[30mb]/[82mb]}
[2013-03-08 13:51:25,000][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][343][63] duration [11.9s], collections
[1]/[11.9s], total [11.9s]/[17m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:51:53,522][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][344][65] duration [28.3s], collections
[2]/[28.3s], total [28.3s]/[17.5m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:52:49,899][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][345][68] duration [37.8s], collections
[3]/[37.8s], total [37.8s]/[18.1m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:54:01,672][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][346][73] duration [1m], collections [5]/[1m],
total [1m]/[19.1m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:54:17,875][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][347][77] duration [44s], collections [4]/[44s],
total [44s]/[19.9m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:55:04,347][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [349785ms] ago,
timed out [319083ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][4guTyorqTUe0mEtoWocYQw][inet[211.110.1.20/211.110.1.20:9300]]{master=false}],
id [420]
[2013-03-08 13:55:22,965][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][348][80] duration [34.4s], collections
[3]/[34.4s], total [34.4s]/[20.4m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:55:22,965][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [337701ms] ago,
timed out [299982ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][4guTyorqTUe0mEtoWocYQw][inet[211.110.1.20/211.110.1.20:9300]]{master=false}],
id [422]
[2013-03-08 13:55:32,168][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][349][83] duration [39.8s], collections
[3]/[39.8s], total [39.8s]/[21.1m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:56:00,358][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [327973ms] ago,
timed out [296685ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][4guTyorqTUe0mEtoWocYQw][inet[211.110.1.20/211.110.1.20:9300]]{master=false}],
id [423]
[2013-03-08 13:57:11,212][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [152115ms] ago,
timed out [0ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node1][dw57v59OQeGn_LuII9z7hg][inet[1.234.83.104/1.234.83.104:9300]]{master=false}],
id [429]
[2013-03-08 13:57:29,750][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][350][90] duration [1.6m], collections [7]/[1.6m],
total [1.6m]/[22.8m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 13:57:39,153][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][351][92] duration [27.9s], collections
[2]/[27.9s], total [27.9s]/[23.2m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [3.1mb]->[3.1mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [30mb]->[30mb]/[82mb]}
[2013-03-08 14:01:35,807][WARN ][transport.netty ] [bingo_node0]
exception caught on transport layer [[id: 0x04281a69, /211.110.1.20:40478
=> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.compress.BufferRecycler.allocEncodingHash(BufferRecycler.java:95)
at
org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java:72)
at
org.elasticsearch.common.compress.lzf.LZFCompressedStreamOutput.(LZFCompressedStreamOutput.java:42)
at
org.elasticsearch.common.compress.lzf.LZFCompressor.streamOutput(LZFCompressor.java:133)
at
org.elasticsearch.common.io.stream.CachedStreamOutput$Entry.handles(CachedStreamOutput.java:72)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:83)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:67)
at
org.elasticsearch.cluster.action.index.NodeAliasesUpdatedAction$NodeAliasesUpdatedTransportHandler.messageReceived(NodeAliasesUpdatedAction.java:117)
at
org.elasticsearch.cluster.action.index.NodeAliasesUpdatedAction$NodeAliasesUpdatedTransportHandler.messageReceived(NodeAliasesUpdatedAction.java:105)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[2013-03-08 14:05:37,845][WARN ][transport.netty ] [bingo_node0]
exception caught on transport layer [[id: 0xa7dd9a40, /211.110.1.20:40500
=> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 14:05:19,247][WARN ][transport.netty ] [bingo_node0]
exception caught on transport layer [[id: 0xc2f655a6, /211.110.1.20:40498
=> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 14:29:23,624][INFO ][node ] [bingo_node0]
{0.20.5}[19651]: initializing ...
[2013-03-08 14:29:23,642][INFO ][plugins ] [bingo_node0]
loaded [], sites [head]
[2013-03-08 14:29:26,042][INFO ][node ] [bingo_node0]
{0.20.5}[19651]: initialized
[2013-03-08 14:29:26,043][INFO ][node ] [bingo_node0]
{0.20.5}[19651]: starting ...
[2013-03-08 14:29:26,165][INFO ][transport ] [bingo_node0]
bound_address {inet[/1.234.83.149:9300]}, publish_address
{inet[/1.234.83.149:9300]}
[2013-03-08 14:29:29,272][INFO ][cluster.service ] [bingo_node0]
new_master
[bingo_node0][auZIU6m1Rb2Cf0GOe_YYhQ][inet[/1.234.83.149:9300]]{master=true},
reason: zen-disco-join (elected_as_master)
[2013-03-08 14:29:29,279][INFO ][discovery ] [bingo_node0]
bingo_dist/auZIU6m1Rb2Cf0GOe_YYhQ
[2013-03-08 14:29:29,294][INFO ][http ] [bingo_node0]
bound_address {inet[/1.234.83.149:9200]}, publish_address
{inet[/1.234.83.149:9200]}
[2013-03-08 14:29:29,294][INFO ][node ] [bingo_node0]
{0.20.5}[19651]: started
[2013-03-08 14:29:51,057][INFO ][cluster.service ] [bingo_node0]
added
{[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false},},
reason: zen-disco-receive(join from
node[[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}])
[2013-03-08 14:30:11,625][INFO ][cluster.service ] [bingo_node0]
added
{[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false},},
reason: zen-disco-receive(join from
node[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}])
[2013-03-08 14:30:28,377][INFO ][cluster.service ] [bingo_node0]
added
{[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true},},
reason: zen-disco-receive(join from
node[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}])
[2013-03-08 14:30:28,895][INFO ][gateway ] [bingo_node0]
recovered [1] indices into cluster_state
[2013-03-08 14:41:56,724][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][738][1] duration [10.1s], collections
[1]/[11.1s], total [10.1s]/[10.2s], memory [7.1gb]->[6.6gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[102.9mb]->[20.1mb]/[532.5mb]}{[Par Survivor Space]
[66.5mb]->[0b]/[66.5mb]}{[CMS Old Gen] [6.9gb]->[6.6gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:42:13,375][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][741][2] duration [13.9s], collections
[1]/[14.6s], total [13.9s]/[24.1s], memory [7.1gb]->[7.1gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[252.8mb]->[11.5mb]/[532.5mb]}{[Par Survivor Space]
[66.5mb]->[0b]/[66.5mb]}{[CMS Old Gen] [6.8gb]->[7gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:42:31,243][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][743][3] duration [16.7s], collections
[1]/[16.8s], total [16.7s]/[40.9s], memory [7.5gb]->[7.3gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[507.6mb]->[232.9mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[0b]/[66.5mb]}{[CMS Old Gen] [7gb]->[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]->[29mb]/[82mb]}
[2013-03-08 14:42:50,510][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][744][4] duration [18.4s], collections
[1]/[19.2s], total [18.4s]/[59.3s], memory [7.3gb]->[7.5gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[232.9mb]->[391.9mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]->[29mb]/[82mb]}
[2013-03-08 14:43:09,926][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][745][5] duration [18.6s], collections
[1]/[19.4s], total [18.6s]/[1.2m], memory [7.5gb]->[7.6gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[391.9mb]->[500mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]->[29mb]/[82mb]}
[2013-03-08 14:43:29,541][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][746][6] duration [19s], collections [1]/[19.6s],
total [19s]/[1.6m], memory [7.6gb]->[7.6gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[500mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[10.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:43:49,239][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][747][7] duration [19.2s], collections
[1]/[19.6s], total [19.2s]/[1.9m], memory [7.6gb]->[7.2gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[80.8mb]/[532.5mb]}{[Par Survivor Space]
[10.8mb]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:44:09,094][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][749][8] duration [18.3s], collections
[1]/[18.7s], total [18.3s]/[2.2m], memory [7.6gb]->[7.4gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[338.9mb]/[532.5mb]}{[Par Survivor Space]
[7.9mb]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:44:26,939][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][750][9] duration [16.9s], collections
[1]/[17.9s], total [16.9s]/[2.5m], memory [7.4gb]->[7.6gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[338.9mb]->[472.9mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]->[29mb]/[82mb]}
[2013-03-08 14:44:46,945][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][751][10] duration [19.3s], collections [1]/[20s],
total [19.3s]/[2.8m], memory [7.6gb]->[7.6gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[472.9mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[272kb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:45:06,752][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][752][11] duration [19.1s], collections
[1]/[19.7s], total [19.1s]/[3.1m], memory [7.6gb]->[7.4gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[244.9mb]/[532.5mb]}{[Par Survivor Space]
[272kb]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:45:27,857][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][753][12] duration [20.3s], collections
[1]/[21.2s], total [20.3s]/[3.5m], memory [7.4gb]->[7.5gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[244.9mb]->[416.2mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]->[29mb]/[82mb]}
[2013-03-08 14:45:49,213][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][754][13] duration [20.7s], collections
[1]/[21.3s], total [20.7s]/[3.8m], memory [7.5gb]->[7.6gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[416.2mb]->[496.8mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[0b]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm Gen]
[29mb]->[29mb]/[82mb]}
[2013-03-08 14:46:10,818][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][755][14] duration [21s], collections [1]/[21.6s],
total [21s]/[4.2m], memory [7.6gb]->[7.6gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[496.8mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[0b]->[13.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:46:32,436][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][756][15] duration [21.2s], collections
[1]/[21.6s], total [21.2s]/[4.5m], memory [7.6gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[13.6mb]->[21.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:46:53,478][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][757][16] duration [20.7s], collections [1]/[21s],
total [20.7s]/[4.9m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[21.8mb]->[42.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:47:12,208][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][758][17] duration [18.5s], collections
[1]/[18.7s], total [18.5s]/[5.2m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[42.8mb]->[54.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:47:33,478][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][759][18] duration [21.1s], collections
[1]/[21.2s], total [21.1s]/[5.5m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[54.8mb]->[60.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:47:55,086][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][760][19] duration [21.5s], collections
[1]/[21.6s], total [21.5s]/[5.9m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[60.6mb]->[63mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:48:16,660][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][761][20] duration [21.5s], collections
[1]/[21.5s], total [21.5s]/[6.2m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[63mb]->[64.5mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:48:36,173][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][762][21] duration [19.5s], collections
[1]/[19.5s], total [19.5s]/[6.6m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[64.5mb]->[65.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:48:55,328][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][763][22] duration [19.1s], collections
[1]/[19.1s], total [19.1s]/[6.9m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.1mb]->[65.6mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:49:14,624][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][764][23] duration [19.2s], collections
[1]/[19.2s], total [19.2s]/[7.2m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.6mb]->[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:49:14,625][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [57965ms] ago,
timed out [19296ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2203]
[2013-03-08 14:49:34,278][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][765][24] duration [19.4s], collections
[1]/[19.4s], total [19.4s]/[7.5m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]->[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:49:56,381][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][766][25] duration [22s], collections [1]/[22.1s],
total [22s]/[7.9m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]->[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS Perm
Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:50:18,383][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][767][26] duration [22.2s], collections
[1]/[22.2s], total [22.2s]/[8.3m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]->[65.5mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:50:18,599][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [44321ms] ago,
timed out [215ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2208]
[2013-03-08 14:50:18,599][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [63975ms] ago,
timed out [22218ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2207]
[2013-03-08 14:50:40,412][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][768][27] duration [21.8s], collections
[1]/[21.8s], total [21.8s]/[8.6m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.5mb]->[65mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:51:02,454][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][769][28] duration [22.2s], collections
[1]/[22.2s], total [22.2s]/[9m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools
{[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65mb]->[65.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:51:02,454][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [66073ms] ago,
timed out [22041ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2210]
[2013-03-08 14:51:24,756][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][770][29] duration [22.2s], collections
[1]/[22.3s], total [22.2s]/[9.4m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.4mb]->[65.8mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:51:24,971][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [44559ms] ago,
timed out [215ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2212]
[2013-03-08 14:51:47,120][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][771][30] duration [22.3s], collections
[1]/[22.3s], total [22.3s]/[9.7m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.8mb]->[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:52:06,644][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [64191ms] ago,
timed out [19523ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2214]
[2013-03-08 14:52:59,425][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][772][32] duration [31.3s], collections
[2]/[31.3s], total [31.3s]/[10.3m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:53:20,698][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][773][35] duration [1m], collections [3]/[1m],
total [1m]/[11.3m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:53:39,328][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][774][36] duration [18.8s], collections
[1]/[18.8s], total [18.8s]/[11.6m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:54:13,136][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [73498ms] ago,
timed out [33808ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2218]
[2013-03-08 14:54:22,528][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][775][38] duration [33.8s], collections
[2]/[33.8s], total [33.8s]/[12.2m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:55:15,161][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [95834ms] ago,
timed out [21282ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2221]
[2013-03-08 14:55:15,161][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [95833ms] ago,
timed out [52633ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2222]
[2013-03-08 14:55:49,218][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][776][43] duration [1.4m], collections [5]/[1.4m],
total [1.4m]/[13.6m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:56:10,588][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][777][45] duration [33.2s], collections
[2]/[33.2s], total [33.2s]/[14.1m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:56:10,589][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [55214ms] ago,
timed out [21370ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2225]
[2013-03-08 14:56:44,149][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [100774ms] ago,
timed out [33367ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}],
id [2223]
[2013-03-08 14:57:17,844][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][778][48] duration [45.5s], collections
[3]/[45.5s], total [45.5s]/[14.9m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:58:08,143][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [138924ms] ago,
timed out [50299ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2226]
[2013-03-08 14:58:17,395][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [114810ms] ago,
timed out [9252ms] ago, action [/gateway/local/started-shards/n], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2228]
[2013-03-08 14:58:17,395][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][779][53] duration [1.3m], collections [5]/[1.3m],
total [1.3m]/[16.2m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]->[65.9mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:58:38,929][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][780][54] duration [21.7s], collections
[1]/[21.7s], total [21.7s]/[16.6m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[65.9mb]->[66mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29mb]/[82mb]}
[2013-03-08 14:58:38,929][INFO ][cluster.service ] [bingo_node0]
removed
{[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false},},
reason:
zen-disco-node_failed([bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}),
reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-03-08 14:59:53,034][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [188885ms] ago,
timed out [95636ms] ago, action [/gateway/local/started-shards/n], node
[[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}],
id [2229]
[2013-03-08 14:59:53,034][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][781][58] duration [52.3s], collections
[4]/[52.3s], total [52.3s]/[17.5m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66mb]->[66.4mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29mb]->[29.1mb]/[82mb]}
[2013-03-08 15:00:05,033][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][782][60] duration [33.7s], collections
[2]/[33.7s], total [33.7s]/[18m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.4mb]->[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.1mb]->[29.1mb]/[82mb]}
[2013-03-08 15:00:26,606][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][783][61] duration [21.5s], collections
[1]/[21.5s], total [21.5s]/[18.4m], memory [7.7gb]->[7.7gb]/[7.7gb],
all_pools {[Code Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]->[66.2mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.1mb]->[29.1mb]/[82mb]}
[2013-03-08 15:00:38,655][DEBUG][action.search.type ] [bingo_node0]
[20130305][1], node[nXyQPEDuT4qXlUlXcW2ALQ], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@63012d7c]
org.elasticsearch.transport.NodeDisconnectedException:
[bingo_node3][inet[/211.110.1.20:9300]][search/phase/query] disconnected
[2013-03-08 15:01:00,382][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [150513ms] ago,
timed out [119727ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2231]
[2013-03-08 15:01:00,382][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][784][62] duration [12s], collections [1]/[12s],
total [12s]/[18.6m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.2mb]->[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.1mb]->[29.2mb]/[82mb]}
[2013-03-08 15:01:33,692][WARN ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][785][65] duration [55s], collections [3]/[55s],
total [55s]/[19.5m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.3mb]->[66.1mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.2mb]->[29.2mb]/[82mb]}
[2013-03-08 15:01:43,058][INFO ][monitor.jvm ] [bingo_node0]
[gc][ConcurrentMarkSweep][786][66] duration [9.3s], collections [1]/[9.3s],
total [9.3s]/[19.7m], memory [7.7gb]->[7.7gb]/[7.7gb], all_pools {[Code
Cache] [1.3mb]->[1.3mb]/[48mb]}{[Par Eden Space]
[532.5mb]->[532.5mb]/[532.5mb]}{[Par Survivor Space]
[66.1mb]->[66.3mb]/[66.5mb]}{[CMS Old Gen] [7.1gb]->[7.1gb]/[7.1gb]}{[CMS
Perm Gen] [29.2mb]->[29.2mb]/[82mb]}
[2013-03-08 15:02:04,355][INFO ][cluster.service ] [bingo_node0]
removed
{[bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false},},
reason:
zen-disco-node_failed([bingo_node1][5OkiRwVySNukFAxK8J68cg][inet[/1.234.83.104:9300]]{master=false}),
reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-03-08 15:02:04,356][DEBUG][action.search.type ] [bingo_node0]
[20130305][3], node[5OkiRwVySNukFAxK8J68cg], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@63012d7c]
org.elasticsearch.transport.NodeDisconnectedException:
[bingo_node1][inet[/1.234.83.104:9300]][search/phase/query] disconnected
[2013-03-08 15:02:49,890][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [250747ms] ago,
timed out [176641ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2232]
[2013-03-08 15:06:10,473][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [310092ms] ago,
timed out [276781ms] ago, action [/gateway/local/started-shards/n], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2237]
[2013-03-08 15:08:05,076][WARN ][transport ] [bingo_node0]
Received response for a request that has timed out, sent [492041ms] ago,
timed out [446421ms] ago, action [discovery/zen/fd/ping], node
[[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}],
id [2236]
[2013-03-08 15:18:02,663][WARN ][transport.netty ] [bingo_node0]
exception caught on transport layer [[id: 0x4f47082e, /211.110.1.24:35454
=> /1.234.83.149:9300]], closing connection
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.compress.BufferRecycler.allocEncodingHash(BufferRecycler.java:95)
at
org.elasticsearch.common.compress.lzf.ChunkEncoder.(ChunkEncoder.java:72)
at
org.elasticsearch.common.compress.lzf.LZFCompressedStreamOutput.(LZFCompressedStreamOutput.java:42)
at
org.elasticsearch.common.compress.lzf.LZFCompressor.streamOutput(LZFCompressor.java:133)
at
org.elasticsearch.common.io.stream.CachedStreamOutput$Entry.handles(CachedStreamOutput.java:72)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:83)
at
org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:67)
at
org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler.messageReceived(MasterFaultDetection.java:387)
at
org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPingRequestHandler.messageReceived(MasterFaultDetection.java:362)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[2013-03-08 15:18:48,782][INFO ][cluster.service ] [bingo_node0]
removed
{[bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true},},
reason:
zen-disco-node_failed([bingo_node2][hXCZyzPzRRiNjl1KdIRegA][inet[/211.110.1.24:9300]]{master=true}),
reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2013-03-08 15:18:36,721][DEBUG][action.search.type ] [bingo_node0]
[20130305][0], node[hXCZyzPzRRiNjl1KdIRegA], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@63012d7c]
org.elasticsearch.transport.NodeDisconnectedException:
[bingo_node2][inet[/211.110.1.24:9300]][search/phase/query] disconnected
[2013-03-08 15:31:43,246][WARN ][index.engine.robin ] [bingo_node0]
[20130305][2] failed engine
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 15:32:11,408][WARN ][netty.channel.DefaultChannelPipeline] An
exception was thrown by a user handler while handling an exception event
([id: 0xc26ff33f, /1.234.83.149:45454 => /1.234.83.149:9300] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
[2013-03-08 15:37:53,195][INFO ][node ] [bingo_node0]
{0.20.5}[20685]: initializing ...
[2013-03-08 15:37:53,199][INFO ][plugins ] [bingo_node0]
loaded [], sites [head]
[2013-03-08 15:37:55,111][INFO ][node ] [bingo_node0]
{0.20.5}[20685]: initialized
[2013-03-08 15:37:55,111][INFO ][node ] [bingo_node0]
{0.20.5}[20685]: starting ...
[2013-03-08 15:37:55,207][INFO ][transport ] [bingo_node0]
bound_address {inet[/1.234.83.149:9300]}, publish_address
{inet[/1.234.83.149:9300]}
[2013-03-08 15:37:59,062][WARN ][discovery.zen.ping.unicast] [bingo_node0]
failed to send ping to [[#zen_unicast_3#][inet[/211.110.1.24:9300]]]
org.elasticsearch.transport.ReceiveTimeoutTransportException:
[][inet[/211.110.1.24:9300]][discovery/zen/unicast] request_id [1] timed
out after [3752ms]
at
org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2013-03-08 15:37:59,811][INFO ][cluster.service ] [bingo_node0]
new_master
[bingo_node0][FgOry_9FTsynZMJXZVnkwA][inet[/1.234.83.149:9300]]{master=true},
reason: zen-disco-join (elected_as_master)
[2013-03-08 15:37:59,818][INFO ][discovery ] [bingo_node0]
bingo_dist/FgOry_9FTsynZMJXZVnkwA
[2013-03-08 15:37:59,834][INFO ][http ] [bingo_node0]
bound_address {inet[/1.234.83.149:9200]}, publish_address
{inet[/1.234.83.149:9200]}
[2013-03-08 15:37:59,834][INFO ][node ] [bingo_node0]
{0.20.5}[20685]: started
[2013-03-08 15:38:03,840][INFO ][cluster.service ] [bingo_node0]
added
{[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false},},
reason: zen-disco-receive(join from
node[[bingo_node3][nXyQPEDuT4qXlUlXcW2ALQ][inet[/211.110.1.20:9300]]{master=false}])
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.