TTL Load Problems

Drew_Daugherty · October 15, 2012, 8:34pm

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+ documents
in it. The index has 6 shards. Disk size consumed by this index is around
6.5 TB (excluding replicas). We set a 30 day default TTL on the index
mapping when it was created. On the day the TTL was set to expire, there
was sudden large spike in load and swap activity that seems to have brought
down nodes in the cluster. The logs contained OOMEs and unfortunately one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL
settings with very large indexes? We are now considering going to a scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

mvg · October 15, 2012, 9:46pm

Hi Drew,

I think using TTL is fine. This could also have happened if a normal
large delete was triggered. Can you share your hardware details? Also
would be great if we can see some stacktraces from the log. Can you
also share your index segments with the indices segment api
(http://localhost:9200/_segments)?

The index consists out of segments, which are read only (expects for
deletes). Normally segments get added to the index when data is added
or updated. When documents get deleted they initially get marked as
deleted. After some time these segments get merged into a bigger
segment and then deleted documents are purged. Merging is managed by a
MergePolicy. Merging is an expensive operation. Luckily a lot can be
configured regarding to merging.

ES supports throttling for merging, but this isn't enabled by default.
Enabling this allows you to use less system resources for merging.
Merges are then slower but then at least don't bring the system down:

The MergePolicy can also be configured. Sometimes it makes sense to
not merge large segments. Maybe this also applies in your case. The
options for merging can be found:

Before changing it is wise to check the output of the indices segment api.

Martijn

On 15 October 2012 22:34, Drew Daugherty djdaugherty612@gmail.com wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+ documents
in it. The index has 6 shards. Disk size consumed by this index is around
6.5 TB (excluding replicas). We set a 30 day default TTL on the index
mapping when it was created. On the day the TTL was set to expire, there
was sudden large spike in load and swap activity that seems to have brought
down nodes in the cluster. The logs contained OOMEs and unfortunately one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL
settings with very large indexes? We are now considering going to a scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen

--

Drew_Daugherty · October 15, 2012, 11:35pm

Thanks Martijn,

I have attached the output from the segments command. Hardware consists of
6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of heap
memory to the ES JVM. Log entries from the master are below.

-drew

[2012-10-13 00:09:21,820][INFO ][monitor.jvm ] [esn-03]
[gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], total
[728ms]/[53.4m], memory [10.5gb]->[10.4gb]/[11.9gb], all_pools {[Code
Cache] [9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[211.6mb]->[36.7mb]/[216.3mb]}{[Par Survivor Space]
[26.4mb]->[27mb]/[27mb]}{[CMS Old Gen] [10.2gb]->[10.3gb]/[11.7gb]}{[CMS
Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 04:21:59,041][WARN ][monitor.jvm ] [esn-03]
[gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], total
[2.5s]/[54.3m], memory [10.4gb]->[10.4gb]/[11.9gb], all_pools {[Code Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space] [38.5mb]->[2.6mb]/[216.3mb]}{[Par
Survivor Space] [26.7mb]->[26.9mb]/[27mb]}{[CMS Old Gen]
[10.3gb]->[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 09:51:00,359][WARN ][transport ] [esn-03]
Received response for a request that has timed out, sent [163943ms] ago,
timed out [133943ms] ago, action [discovery/zen/fd/ping], node
[[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id
[9346581]
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:55:58,460][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed engine
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:56:02,853][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw],
[R], s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked
shard as started, but shard have not been created, mark shard as failed]
[2012-10-13 09:56:03,035][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed to flush after setting shard to inactive
org.elasticsearch.index.engine.FlushFailedEngineException: [messages2][0]
Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
... 12 more
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-02][inet[/10.8.2.47:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:39,288][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ],
[P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg],
[R], s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked
shard as started, but shard have not been created, mark shard as failed]
[2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:20,317][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-05][inet[/10.8.2.50:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen wrote:

Hi Drew,

I think using TTL is fine. This could also have happened if a normal
large delete was triggered. Can you share your hardware details? Also
would be great if we can see some stacktraces from the log. Can you
also share your index segments with the indices segment api
(http://localhost:9200/_segments)?

The index consists out of segments, which are read only (expects for
deletes). Normally segments get added to the index when data is added
or updated. When documents get deleted they initially get marked as
deleted. After some time these segments get merged into a bigger
segment and then deleted documents are purged. Merging is managed by a
MergePolicy. Merging is an expensive operation. Luckily a lot can be
configured regarding to merging.

ES supports throttling for merging, but this isn't enabled by default.
Enabling this allows you to use less system resources for merging.
Merges are then slower but then at least don't bring the system down:
Elasticsearch Platform — Find real-time answers at scale | Elastic

The MergePolicy can also be configured. Sometimes it makes sense to
not merge large segments. Maybe this also applies in your case. The
options for merging can be found:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Before changing it is wise to check the output of the indices segment api.

Martijn

On 15 October 2012 22:34, Drew Daugherty <djdaugh...@gmail.com<javascript:>>
wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents
in it. The index has 6 shards. Disk size consumed by this index is
around
6.5 TB (excluding replicas). We set a 30 day default TTL on the index
mapping when it was created. On the day the TTL was set to expire,
there
was sudden large spike in load and swap activity that seems to have
brought
down nodes in the cluster. The logs contained OOMEs and unfortunately
one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use
TTL
settings with very large indexes? We are now considering going to a
scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen

--

Drew_Daugherty · October 15, 2012, 11:37pm

Sorry I meant 12GB for heap memory.

-drew

On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote:

Thanks Martijn,

I have attached the output from the segments command. Hardware consists of
6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of heap
memory to the ES JVM. Log entries from the master are below.

-drew

[2012-10-13 00:09:21,820][INFO ][monitor.jvm ] [esn-03]
[gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], total
[728ms]/[53.4m], memory [10.5gb]->[10.4gb]/[11.9gb], all_pools {[Code
Cache] [9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[211.6mb]->[36.7mb]/[216.3mb]}{[Par Survivor Space]
[26.4mb]->[27mb]/[27mb]}{[CMS Old Gen] [10.2gb]->[10.3gb]/[11.7gb]}{[CMS
Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 04:21:59,041][WARN ][monitor.jvm ] [esn-03]
[gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], total
[2.5s]/[54.3m], memory [10.4gb]->[10.4gb]/[11.9gb], all_pools {[Code Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space] [38.5mb]->[2.6mb]/[216.3mb]}{[Par
Survivor Space] [26.7mb]->[26.9mb]/[27mb]}{[CMS Old Gen]
[10.3gb]->[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 09:51:00,359][WARN ][transport ] [esn-03]
Received response for a request that has timed out, sent [163943ms] ago,
timed out [133943ms] ago, action [discovery/zen/fd/ping], node
[[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id
[9346581]
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:55:58,460][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed engine
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:56:02,853][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw],
[R], s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked
shard as started, but shard have not been created, mark shard as failed]
[2012-10-13 09:56:03,035][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed to flush after setting shard to inactive
org.elasticsearch.index.engine.FlushFailedEngineException: [messages2][0]
Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
... 12 more
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-02][inet[/10.8.2.47:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:39,288][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ],
[P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg],
[R], s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked
shard as started, but shard have not been created, mark shard as failed]
[2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:20,317][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA],
[R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap space]]]
[2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-05][inet[/10.8.2.50:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen wrote:

Hi Drew,

I think using TTL is fine. This could also have happened if a normal
large delete was triggered. Can you share your hardware details? Also
would be great if we can see some stacktraces from the log. Can you
also share your index segments with the indices segment api
(http://localhost:9200/_segments)?

The index consists out of segments, which are read only (expects for
deletes). Normally segments get added to the index when data is added
or updated. When documents get deleted they initially get marked as
deleted. After some time these segments get merged into a bigger
segment and then deleted documents are purged. Merging is managed by a
MergePolicy. Merging is an expensive operation. Luckily a lot can be
configured regarding to merging.

ES supports throttling for merging, but this isn't enabled by default.
Enabling this allows you to use less system resources for merging.
Merges are then slower but then at least don't bring the system down:
Elasticsearch Platform — Find real-time answers at scale | Elastic

The MergePolicy can also be configured. Sometimes it makes sense to
not merge large segments. Maybe this also applies in your case. The
options for merging can be found:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Before changing it is wise to check the output of the indices segment
api.

Martijn

On 15 October 2012 22:34, Drew Daugherty djdaugh...@gmail.com wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents
in it. The index has 6 shards. Disk size consumed by this index is
around
6.5 TB (excluding replicas). We set a 30 day default TTL on the index
mapping when it was created. On the day the TTL was set to expire,
there
was sudden large spike in load and swap activity that seems to have
brought
down nodes in the cluster. The logs contained OOMEs and unfortunately
one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use
TTL
settings with very large indexes? We are now considering going to a
scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen

--

mvg · October 16, 2012, 10:24am

From the stacktraces it seems that the OOM occurs after the merging
has completed and during the new view on the index is being opened.
Are using sorting or faceting? This takes usually a lot of memory. Can
you share you nodes info (http://localhost:9200/_nodes/stats?all)?

Perhaps you need to increase the number of nodes or increase the heap
space size.

Martijn

On 16 October 2012 01:37, Drew Daugherty djdaugherty612@gmail.com wrote:

Sorry I meant 12GB for heap memory.

-drew

On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote:

Thanks Martijn,

I have attached the output from the segments command. Hardware consists of
6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of heap
memory to the ES JVM. Log entries from the master are below.

-drew

[2012-10-13 00:09:21,820][INFO ][monitor.jvm ] [esn-03]
[gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], total
[728ms]/[53.4m], memory [10.5gb]->[10.4gb]/[11.9gb], all_pools {[Code Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[211.6mb]->[36.7mb]/[216.3mb]}{[Par Survivor Space]
[26.4mb]->[27mb]/[27mb]}{[CMS Old Gen] [10.2gb]->[10.3gb]/[11.7gb]}{[CMS
Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 04:21:59,041][WARN ][monitor.jvm ] [esn-03]
[gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], total
[2.5s]/[54.3m], memory [10.4gb]->[10.4gb]/[11.9gb], all_pools {[Code Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space] [38.5mb]->[2.6mb]/[216.3mb]}{[Par
Survivor Space] [26.7mb]->[26.9mb]/[27mb]}{[CMS Old Gen]
[10.3gb]->[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 09:51:00,359][WARN ][transport ] [esn-03]
Received response for a request that has timed out, sent [163943ms] ago,
timed out [133943ms] ago, action [discovery/zen/fd/ping], node
[[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id
[9346581]
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:55:58,460][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed engine
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:56:02,853][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R],
s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked
shard as started, but shard have not been created, mark shard as failed]
[2012-10-13 09:56:03,035][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed to flush after setting shard to inactive
org.elasticsearch.index.engine.FlushFailedEngineException: [messages2][0]
Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837)
at
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296)
at
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
... 12 more
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-02][inet[/10.8.2.47:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:56:39,288][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R],
s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked
shard as started, but shard have not been created, mark shard as failed]
[2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:20,317][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]
org.elasticsearch.transport.RemoteTransportException:
[esn-05][inet[/10.8.2.50:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]
org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at
org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen wrote:

Hi Drew,

I think using TTL is fine. This could also have happened if a normal
large delete was triggered. Can you share your hardware details? Also
would be great if we can see some stacktraces from the log. Can you
also share your index segments with the indices segment api
(http://localhost:9200/_segments)?

The index consists out of segments, which are read only (expects for
deletes). Normally segments get added to the index when data is added
or updated. When documents get deleted they initially get marked as
deleted. After some time these segments get merged into a bigger
segment and then deleted documents are purged. Merging is managed by a
MergePolicy. Merging is an expensive operation. Luckily a lot can be
configured regarding to merging.

ES supports throttling for merging, but this isn't enabled by default.
Enabling this allows you to use less system resources for merging.
Merges are then slower but then at least don't bring the system down:
Elasticsearch Platform — Find real-time answers at scale | Elastic

The MergePolicy can also be configured. Sometimes it makes sense to
not merge large segments. Maybe this also applies in your case. The
options for merging can be found:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Before changing it is wise to check the output of the indices segment
api.

Martijn

On 15 October 2012 22:34, Drew Daugherty djdaugh...@gmail.com wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents
in it. The index has 6 shards. Disk size consumed by this index is
around
6.5 TB (excluding replicas). We set a 30 day default TTL on the index
mapping when it was created. On the day the TTL was set to expire,
there
was sudden large spike in load and swap activity that seems to have
brought
down nodes in the cluster. The logs contained OOMEs and unfortunately
one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use
TTL
settings with very large indexes? We are now considering going to a
scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen

--

--
Met vriendelijke groet,

Martijn van Groningen

--

Drew_Daugherty · October 16, 2012, 3:18pm

See attached for stats all output after a fresh cluster restart (it locked
up again last night - OOMEs). We are currently using different settings to
cut down the size of our indexes.

-drew

On Tuesday, October 16, 2012 4:24:51 AM UTC-6, Martijn v Groningen wrote:

From the stacktraces it seems that the OOM occurs after the merging
has completed and during the new view on the index is being opened.
Are using sorting or faceting? This takes usually a lot of memory. Can
you share you nodes info (http://localhost:9200/_nodes/stats?all)?

Perhaps you need to increase the number of nodes or increase the heap
space size.

Martijn

On 16 October 2012 01:37, Drew Daugherty <djdaugh...@gmail.com<javascript:>>
wrote:

Sorry I meant 12GB for heap memory.

-drew

On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote:

Thanks Martijn,

I have attached the output from the segments command. Hardware consists
of
6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of
heap
memory to the ES JVM. Log entries from the master are below.

-drew

[2012-10-13 00:09:21,820][INFO ][monitor.jvm ] [esn-03]
[gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s],
total
[728ms]/[53.4m], memory [10.5gb]->[10.4gb]/[11.9gb], all_pools {[Code
Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[211.6mb]->[36.7mb]/[216.3mb]}{[Par Survivor Space]
[26.4mb]->[27mb]/[27mb]}{[CMS Old Gen]
[10.2gb]->[10.3gb]/[11.7gb]}{[CMS
Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 04:21:59,041][WARN ][monitor.jvm ] [esn-03]
[gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s],
total
[2.5s]/[54.3m], memory [10.4gb]->[10.4gb]/[11.9gb], all_pools {[Code
Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[38.5mb]->[2.6mb]/[216.3mb]}{[Par
Survivor Space] [26.7mb]->[26.9mb]/[27mb]}{[CMS Old Gen]
[10.3gb]->[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 09:51:00,359][WARN ][transport ] [esn-03]
Received response for a request that has timed out, sent [163943ms]
ago,
timed out [133943ms] ago, action [discovery/zen/fd/ping], node

[[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id

[9346581]
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg],
[R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:55:58,460][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed engine
java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at

org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at 
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at 
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at 
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at 
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at 
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) 
at 
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at 
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at 
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
at 
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at 
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at 
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) 
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw],
[R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:56:02,853][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw],
[R],
s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true}
marked
shard as started, but shard have not been created, mark shard as
failed]
[2012-10-13 09:56:03,035][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed to flush after setting shard to inactive
org.elasticsearch.index.engine.FlushFailedEngineException:
[messages2][0]
Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837)
at 
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at 
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at 
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) 
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at 
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at 
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at 
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at 
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at 
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) 
at 
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at 
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at 
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
... 12 more 
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]

org.elasticsearch.transport.RemoteTransportException:
[esn-02][inet[/10.8.2.47:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]

org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:56:39,288][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ],
[P],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg],
[R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg],
[R],
s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true}
marked
shard as started, but shard have not been created, mark shard as
failed]
[2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.index.IndexShardMissingException: [messages2][0]
missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:20,317][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA],
[R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap
space]]]
[2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.index.IndexShardMissingException: [messages2][0]
missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-05][inet[/10.8.2.50:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][0] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]

org.elasticsearch.index.IndexShardMissingException: [messages2][0]
missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]

org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen
wrote:

Hi Drew,

I think using TTL is fine. This could also have happened if a normal
large delete was triggered. Can you share your hardware details? Also
would be great if we can see some stacktraces from the log. Can you
also share your index segments with the indices segment api
(http://localhost:9200/_segments)?

The index consists out of segments, which are read only (expects for
deletes). Normally segments get added to the index when data is added
or updated. When documents get deleted they initially get marked as
deleted. After some time these segments get merged into a bigger
segment and then deleted documents are purged. Merging is managed by a
MergePolicy. Merging is an expensive operation. Luckily a lot can be
configured regarding to merging.

ES supports throttling for merging, but this isn't enabled by default.
Enabling this allows you to use less system resources for merging.
Merges are then slower but then at least don't bring the system down:
Elasticsearch Platform — Find real-time answers at scale | Elastic

The MergePolicy can also be configured. Sometimes it makes sense to
not merge large segments. Maybe this also applies in your case. The
options for merging can be found:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Before changing it is wise to check the output of the indices segment
api.

Martijn

On 15 October 2012 22:34, Drew Daugherty djdaugh...@gmail.com
wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents
in it. The index has 6 shards. Disk size consumed by this index is
around
6.5 TB (excluding replicas). We set a 30 day default TTL on the
index
mapping when it was created. On the day the TTL was set to expire,
there
was sudden large spike in load and swap activity that seems to have
brought
down nodes in the cluster. The logs contained OOMEs and
unfortunately
one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to
use
TTL
settings with very large indexes? We are now considering going to a
scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen
--
--
Met vriendelijke groet,

Martijn van Groningen

--

Drew_Daugherty · October 16, 2012, 4:54pm

Martijn,

See attached for all stats after things are up and indexing documents.

-drew

On Tuesday, October 16, 2012 9:18:46 AM UTC-6, Drew Daugherty wrote:

See attached for stats all output after a fresh cluster restart (it locked
up again last night - OOMEs). We are currently using different settings to
cut down the size of our indexes.

-drew

On Tuesday, October 16, 2012 4:24:51 AM UTC-6, Martijn v Groningen wrote:
From the stacktraces it seems that the OOM occurs after the merging
has completed and during the new view on the index is being opened.
Are using sorting or faceting? This takes usually a lot of memory. Can
you share you nodes info (http://localhost:9200/_nodes/stats?all)?

Perhaps you need to increase the number of nodes or increase the heap
space size.

Martijn

On 16 October 2012 01:37, Drew Daugherty djdaugh...@gmail.com wrote:

Sorry I meant 12GB for heap memory.

-drew

On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote:

Thanks Martijn,

I have attached the output from the segments command. Hardware
consists of
6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of
heap
memory to the ES JVM. Log entries from the master are below.

-drew

[2012-10-13 00:09:21,820][INFO ][monitor.jvm ] [esn-03]
[gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s],
total
[728ms]/[53.4m], memory [10.5gb]->[10.4gb]/[11.9gb], all_pools {[Code
Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[211.6mb]->[36.7mb]/[216.3mb]}{[Par Survivor Space]
[26.4mb]->[27mb]/[27mb]}{[CMS Old Gen]
[10.2gb]->[10.3gb]/[11.7gb]}{[CMS
Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 04:21:59,041][WARN ][monitor.jvm ] [esn-03]
[gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s],
total
[2.5s]/[54.3m], memory [10.4gb]->[10.4gb]/[11.9gb], all_pools {[Code
Cache]
[9.5mb]->[9.5mb]/[48mb]}{[Par Eden Space]
[38.5mb]->[2.6mb]/[216.3mb]}{[Par
Survivor Space] [26.7mb]->[26.9mb]/[27mb]}{[CMS Old Gen]
[10.3gb]->[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]->[47mb]/[82mb]}
[2012-10-13 09:51:00,359][WARN ][transport ] [esn-03]
Received response for a request that has timed out, sent [163943ms]
ago,
timed out [133943ms] ago, action [discovery/zen/fd/ping], node

[[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id

[9346581]
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4],
node[q6Al5C7HQ-a2TJotgg1ojg], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap
space]]]
[2012-10-13 09:55:58,460][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed engine
java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at

org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at 
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at 
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at 
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at 
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at 
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) 
at 
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at 
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at 
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
at 
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at 
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at 
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at 
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:55:58,453][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1],
node[5IAi9fteSt6OGHyXH7VrTw], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap
space]]]
[2012-10-13 09:56:02,853][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][1],
node[5IAi9fteSt6OGHyXH7VrTw], [R],
s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true}
marked
shard as started, but shard have not been created, mark shard as
failed]
[2012-10-13 09:56:03,035][WARN ][index.engine.robin ] [esn-03]
[messages2][0] failed to flush after setting shard to inactive
org.elasticsearch.index.engine.FlushFailedEngineException:
[messages2][0]
Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837)
at 
org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219)
at 
org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178)
at 
org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at 
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at 
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.util.packed.Packed64.(Packed64.java:122)
at
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221)
at 
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81)
at 
org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84)
at 
org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83)
at 
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at 
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at 
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at 
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) 
at 
org.apache.lucene.search.SearcherManager.(SearcherManager.java:82)
at 
org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364)
at 
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831)
... 12 more 
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]

org.elasticsearch.transport.RemoteTransportException:
[esn-02][inet[/10.8.2.47:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937]

org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:56:39,288][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][4],
node[Cgz1EWFfS1qM5c4SLa-dyQ], [P],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap
space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2],
node[hLtNHFLNRjStSNdqzk4HNg], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap
space]]]
[2012-10-13 09:57:34,414][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][2],
node[hLtNHFLNRjStSNdqzk4HNg], [R],
s[STARTED], reason [master
[esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true}
marked
shard as started, but shard have not been created, mark shard as
failed]
[2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.index.IndexShardMissingException: [messages2][0]
missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584]

org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:20,317][WARN ][cluster.action.shard ] [esn-03]
received shard failed for [messages2][0],
node[BD4mbA1lSYezrejzi-J-fA], [R],
s[STARTED], reason [engine failure, message [OutOfMemoryError[Java
heap
space]]]
[2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.index.IndexShardMissingException: [messages2][0]
missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-04][inet[/10.8.2.49:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][2] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5]

org.elasticsearch.transport.RemoteTransportException:
[esn-05][inet[/10.8.2.50:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][0] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]

org.elasticsearch.index.IndexShardMissingException: [messages2][0]
missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]

org.elasticsearch.transport.RemoteTransportException:
[esn-01][inet[/10.8.2.46:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][4] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
[2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03]
[messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed
to
execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6]

org.elasticsearch.transport.RemoteTransportException:
[esn-00][inet[/10.8.2.45:9300]][indices/stats/s]
Caused by: org.elasticsearch.index.IndexShardMissingException:
[messages2][1] missing
at

org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145)
at 
org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398)
at 
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384)
at 
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) 
On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen
wrote:

Hi Drew,

I think using TTL is fine. This could also have happened if a normal
large delete was triggered. Can you share your hardware details? Also
would be great if we can see some stacktraces from the log. Can you
also share your index segments with the indices segment api
(http://localhost:9200/_segments)?

The index consists out of segments, which are read only (expects for
deletes). Normally segments get added to the index when data is added
or updated. When documents get deleted they initially get marked as
deleted. After some time these segments get merged into a bigger
segment and then deleted documents are purged. Merging is managed by
a
MergePolicy. Merging is an expensive operation. Luckily a lot can be
configured regarding to merging.

ES supports throttling for merging, but this isn't enabled by
default.
Enabling this allows you to use less system resources for merging.
Merges are then slower but then at least don't bring the system down:
Elasticsearch Platform — Find real-time answers at scale | Elastic

The MergePolicy can also be configured. Sometimes it makes sense to
not merge large segments. Maybe this also applies in your case. The
options for merging can be found:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Before changing it is wise to check the output of the indices segment
api.

Martijn

On 15 October 2012 22:34, Drew Daugherty djdaugh...@gmail.com
wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents
in it. The index has 6 shards. Disk size consumed by this index
is
around
6.5 TB (excluding replicas). We set a 30 day default TTL on the
index
mapping when it was created. On the day the TTL was set to expire,
there
was sudden large spike in load and swap activity that seems to have
brought
down nodes in the cluster. The logs contained OOMEs and
unfortunately
one
of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to
use
TTL
settings with very large indexes? We are now considering going to
a
scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen
--
--
Met vriendelijke groet,

Martijn van Groningen

--

ppearcy · October 16, 2012, 5:11pm

I haven't used TTL before, so am not of much help there, but using a
rolling index approach will definitely be much more efficient. Deletes
bloat your index size until segments are merged and they're expunged. With
rolling indexes you won't have the same bloat.

On Monday, October 15, 2012 2:34:39 PM UTC-6, Drew Daugherty wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents in it. The index has 6 shards. Disk size consumed by this index
is around 6.5 TB (excluding replicas). We set a 30 day default TTL on the
index mapping when it was created. On the day the TTL was set to expire,
there was sudden large spike in load and swap activity that seems to have
brought down nodes in the cluster. The logs contained OOMEs and
unfortunately one of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL
settings with very large indexes? We are now considering going to a scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

mvg · October 16, 2012, 10:08pm

Drew,

What kind of queries / filter executing in general? Are you using the
_parent type in your mapping? Just wondering what else might take that
amount of memory.

Perhaps you can lower the amount segments that are merged at a time.
This can be done by configuring the
'index.merge.policy.max_merge_at_once' option in either the
elasticsearch.yml on each node or via the update indices settings api.
Merging segments that contain deletes takes up more ram than segments
without deletes. In your index most segment do contain deletes. The
default 'index.merge.policy.max_merge_at_once' is 10, Maybe you can
try with a value of 5.

Like ppearcy says, for time based data the rolling index approach
doesn't have the merging overhead.
If that is possible for you that would be a better solution.

Martijn

On 16 October 2012 19:11, ppearcy ppearcy@gmail.com wrote:

I haven't used TTL before, so am not of much help there, but using a rolling
index approach will definitely be much more efficient. Deletes bloat your
index size until segments are merged and they're expunged. With rolling
indexes you won't have the same bloat.

On Monday, October 15, 2012 2:34:39 PM UTC-6, Drew Daugherty wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents in it. The index has 6 shards. Disk size consumed by this index
is around 6.5 TB (excluding replicas). We set a 30 day default TTL on the
index mapping when it was created. On the day the TTL was set to expire,
there was sudden large spike in load and swap activity that seems to have
brought down nodes in the cluster. The logs contained OOMEs and
unfortunately one of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL
settings with very large indexes? We are now considering going to a scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen

--

Drew_Daugherty · October 17, 2012, 3:30pm

So far I don't think we have had a query load that would impact memory
usage significantly. We are still at the stage of ad hoc querying where
one or two people might be testing things. There has been interest in
using faceting but I haven't yet confirmed it is actually being used. We
are not using _parent type in our mappings.

Thanks for the information about tweaking merge settings. I think we will
try rolling indexes to avoid the overhead.

-drew

On Tuesday, October 16, 2012 4:09:23 PM UTC-6, Martijn v Groningen wrote:

Drew,

What kind of queries / filter executing in general? Are you using the
_parent type in your mapping? Just wondering what else might take that
amount of memory.

Perhaps you can lower the amount segments that are merged at a time.
This can be done by configuring the
'index.merge.policy.max_merge_at_once' option in either the
elasticsearch.yml on each node or via the update indices settings api.
Merging segments that contain deletes takes up more ram than segments
without deletes. In your index most segment do contain deletes. The
default 'index.merge.policy.max_merge_at_once' is 10, Maybe you can
try with a value of 5.

Like ppearcy says, for time based data the rolling index approach
doesn't have the merging overhead.
If that is possible for you that would be a better solution.

Martijn

On 16 October 2012 19:11, ppearcy <ppe...@gmail.com <javascript:>> wrote:

I haven't used TTL before, so am not of much help there, but using a
rolling
index approach will definitely be much more efficient. Deletes bloat
your
index size until segments are merged and they're expunged. With rolling
indexes you won't have the same bloat.

On Monday, October 15, 2012 2:34:39 PM UTC-6, Drew Daugherty wrote:

Hi,

I am using elasticsearch 0.19.8 and have a large index with 300M+
documents in it. The index has 6 shards. Disk size consumed by this
index
is around 6.5 TB (excluding replicas). We set a 30 day default TTL on
the
index mapping when it was created. On the day the TTL was set to
expire,
there was sudden large spike in load and swap activity that seems to
have
brought down nodes in the cluster. The logs contained OOMEs and
unfortunately one of our shards became corrupt and will no longer load
(IndexShardMissingExceptions). Is this a bug? Is it advisable to use
TTL
settings with very large indexes? We are now considering going to a
scheme
where we roll our indexes and drop older ones rather than use TTLs.

-drew

--

--
Met vriendelijke groet,

Martijn van Groningen

--