# TTL Load Problems **URL:** \ **Category:** Elasticsearch\ **Created:** [October 15, 2012, 8:34pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361 "2012-10-15T20:34:39Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Drew\_Daugherty](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@Drew\_Daugherty](https://discuss.elastic.co/u/Drew_Daugherty)\ **Post date:** [October 15, 2012, 8:34pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/1 "2012-10-15T20:34:39Z")
Hi, I am using elasticsearch 0.19.8 and have a large index with 300M+ documents in it. The index has 6 shards. Disk size consumed by this index is around 6.5 TB (excluding replicas). We set a 30 day default TTL on the index mapping when it was created. On the day the TTL was set to expire, there was sudden large spike in load and swap activity that seems to have brought down nodes in the cluster. The logs contained OOMEs and unfortunately one of our shards became corrupt and will no longer load (IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL settings with very large indexes? We are now considering going to a scheme where we roll our indexes and drop older ones rather than use TTLs. -drew -- ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [October 15, 2012, 9:46pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/2 "2012-10-15T21:46:01Z")
Hi Drew, I think using TTL is fine. This could also have happened if a normal large delete was triggered. Can you share your hardware details? Also would be great if we can see some stacktraces from the log. Can you also share your index segments with the indices segment api ([http://localhost:9200/\_segments](http://localhost:9200/_segments))? The index consists out of segments, which are read only (expects for deletes). Normally segments get added to the index when data is added or updated. When documents get deleted they initially get marked as deleted. After some time these segments get merged into a bigger segment and then deleted documents are purged. Merging is managed by a MergePolicy. Merging is an expensive operation. Luckily a lot can be configured regarding to merging. ES supports throttling for merging, but this isn't enabled by default. Enabling this allows you to use less system resources for merging. Merges are then slower but then at least don't bring the system down: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. The MergePolicy can also be configured. Sometimes it makes sense to not merge large segments. Maybe this also applies in your case. The options for merging can be found: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Before changing it is wise to check the output of the indices segment api. Martijn On 15 October 2012 22:34, Drew Daugherty [djdaugherty612@gmail.com](mailto:djdaugherty612@gmail.com) wrote: > Hi, > > I am using elasticsearch 0.19.8 and have a large index with 300M+ documents > in it. The index has 6 shards. Disk size consumed by this index is around > 6.5 TB (excluding replicas). We set a 30 day default TTL on the index > mapping when it was created. On the day the TTL was set to expire, there > was sudden large spike in load and swap activity that seems to have brought > down nodes in the cluster. The logs contained OOMEs and unfortunately one > of our shards became corrupt and will no longer load > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL > settings with very large indexes? We are now considering going to a scheme > where we roll our indexes and drop older ones rather than use TTLs. > > -drew > > -- -- Met vriendelijke groet, Martijn van Groningen -- ---
**Author:** ![Drew\_Daugherty](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@Drew\_Daugherty](https://discuss.elastic.co/u/Drew_Daugherty)\ **Post date:** [October 15, 2012, 11:35pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/3 "2012-10-15T23:35:32Z")
Thanks Martijn, I have attached the output from the segments command. Hardware consists of 6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of heap memory to the ES JVM. Log entries from the master are below. -drew [2012-10-13 00:09:21,820][INFO][monitor.jvm] [esn-03] [gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], total [728ms]/[53.4m], memory [10.5gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code Cache] [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] [211.6mb]-\>[36.7mb]/[216.3mb]}{[Par Survivor Space] [26.4mb]-\>[27mb]/[27mb]}{[CMS Old Gen] [10.2gb]-\>[10.3gb]/[11.7gb]}{[CMS Perm Gen] [47mb]-\>[47mb]/[82mb]} [2012-10-13 04:21:59,041][WARN][monitor.jvm] [esn-03] [gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], total [2.5s]/[54.3m], memory [10.4gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code Cache] [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] [38.5mb]-\>[2.6mb]/[216.3mb]}{[Par Survivor Space] [26.7mb]-\>[26.9mb]/[27mb]}{[CMS Old Gen] [10.3gb]-\>[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]-\>[47mb]/[82mb]} [2012-10-13 09:51:00,359][WARN][transport] [esn-03] Received response for a request that has timed out, sent [163943ms] ago, timed out [133943ms] ago, action [discovery/zen/fd/ping], node [[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id [9346581] [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] [2012-10-13 09:55:58,460][WARN][index.engine.robin] [esn-03] [messages2][0] failed engine java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) at org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) at org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) at org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) at org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) at org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] [2012-10-13 09:56:02,853][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED], reason [master [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked shard as started, but shard have not been created, mark shard as failed] [2012-10-13 09:56:03,035][WARN][index.engine.robin] [esn-03] [messages2][0] failed to flush after setting shard to inactive org.elasticsearch.index.engine.FlushFailedEngineException: [messages2][0] Flush failed at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837) at org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) at org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) at org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) at org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) at org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) at org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) ... 12 more [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] org.elasticsearch.transport.RemoteTransportException: [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][1] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] org.elasticsearch.transport.RemoteTransportException: [esn-02][inet[/10.8.2.47:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][4] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] org.elasticsearch.transport.RemoteTransportException: [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][2] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:56:39,288][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED], reason [master [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked shard as started, but shard have not been created, mark shard as failed] [2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03] [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03] [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] org.elasticsearch.transport.RemoteTransportException: [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][1] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03] [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] org.elasticsearch.transport.RemoteTransportException: [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][2] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03] [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] org.elasticsearch.transport.RemoteTransportException: [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][4] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:20,317][WARN][cluster.action.shard] [esn-03] received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]] [2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03] [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03] [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] org.elasticsearch.transport.RemoteTransportException: [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][2] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] org.elasticsearch.transport.RemoteTransportException: [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][1] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] org.elasticsearch.transport.RemoteTransportException: [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][4] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] org.elasticsearch.transport.RemoteTransportException: [esn-05][inet[/10.8.2.50:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03] [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] org.elasticsearch.transport.RemoteTransportException: [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][4] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] org.elasticsearch.transport.RemoteTransportException: [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] Caused by: org.elasticsearch.index.IndexShardMissingException: [messages2][1] missing at org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen wrote: > Hi Drew, > > I think using TTL is fine. This could also have happened if a normal > large delete was triggered. Can you share your hardware details? Also > would be great if we can see some stacktraces from the log. Can you > also share your index segments with the indices segment api > ([http://localhost:9200/\_segments](http://localhost:9200/_segments))? > > The index consists out of segments, which are read only (expects for > deletes). Normally segments get added to the index when data is added > or updated. When documents get deleted they initially get marked as > deleted. After some time these segments get merged into a bigger > segment and then deleted documents are purged. Merging is managed by a > MergePolicy. Merging is an expensive operation. Luckily a lot can be > configured regarding to merging. > > ES supports throttling for merging, but this isn't enabled by default. > Enabling this allows you to use less system resources for merging. > Merges are then slower but then at least don't bring the system down: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > The MergePolicy can also be configured. Sometimes it makes sense to > not merge large segments. Maybe this also applies in your case. The > options for merging can be found: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge.html) > > Before changing it is wise to check the output of the indices segment api. > > Martijn > > On 15 October 2012 22:34, Drew Daugherty \<[djdaugh...@gmail.com](mailto:djdaugh...@gmail.com)\\> > wrote: > > > Hi, > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > documents > > in it. The index has 6 shards. Disk size consumed by this index is > > around > > 6.5 TB (excluding replicas). We set a 30 day default TTL on the index > > mapping when it was created. On the day the TTL was set to expire, > > there > > was sudden large spike in load and swap activity that seems to have > > brought > > down nodes in the cluster. The logs contained OOMEs and unfortunately > > one > > of our shards became corrupt and will no longer load > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use > > TTL > > settings with very large indexes? We are now considering going to a > > scheme > > where we roll our indexes and drop older ones rather than use TTLs. > > > > -drew > > > > -- > > -- > Met vriendelijke groet, > > Martijn van Groningen -- ---
**Author:** ![Drew\_Daugherty](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@Drew\_Daugherty](https://discuss.elastic.co/u/Drew_Daugherty)\ **Post date:** [October 15, 2012, 11:37pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/4 "2012-10-15T23:37:02Z")
Sorry I meant 12GB for heap memory. -drew On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote: > Thanks Martijn, > > I have attached the output from the segments command. Hardware consists of > 6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of heap > memory to the ES JVM. Log entries from the master are below. > > -drew > > [2012-10-13 00:09:21,820][INFO][monitor.jvm] [esn-03] > [gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], total > [728ms]/[53.4m], memory [10.5gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code > Cache] [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] > [211.6mb]-\>[36.7mb]/[216.3mb]}{[Par Survivor Space] > [26.4mb]-\>[27mb]/[27mb]}{[CMS Old Gen] [10.2gb]-\>[10.3gb]/[11.7gb]}{[CMS > Perm Gen] [47mb]-\>[47mb]/[82mb]} > [2012-10-13 04:21:59,041][WARN][monitor.jvm] [esn-03] > [gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], total > [2.5s]/[54.3m], memory [10.4gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code Cache] > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] [38.5mb]-\>[2.6mb]/[216.3mb]}{[Par > Survivor Space] [26.7mb]-\>[26.9mb]/[27mb]}{[CMS Old Gen] > [10.3gb]-\>[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]-\>[47mb]/[82mb]} > [2012-10-13 09:51:00,359][WARN][transport] [esn-03] > Received response for a request that has timed out, sent [163943ms] ago, > timed out [133943ms] ago, action [discovery/zen/fd/ping], node > [[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id > [9346581] > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], > [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > heap space]]] > [2012-10-13 09:55:58,460][WARN][index.engine.robin] [esn-03] > [messages2][0] failed engine > java.lang.OutOfMemoryError: Java heap space > at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > at > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > at > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > at > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > at > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > at > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > at > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > at > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > at > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > at > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > at > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > at > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > at > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > at > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > at > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > at > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], > [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > heap space]]] > [2012-10-13 09:56:02,853][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], > [R], s[STARTED], reason [master > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked > shard as started, but shard have not been created, mark shard as failed] > [2012-10-13 09:56:03,035][WARN][index.engine.robin] [esn-03] > [messages2][0] failed to flush after setting shard to inactive > org.elasticsearch.index.engine.FlushFailedEngineException: [messages2][0] > Flush failed > at > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837) > at > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > at > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > at > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.OutOfMemoryError: Java heap space > at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > at > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > at > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > at > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > at > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > at > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > at > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > at > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > at > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > at > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > at > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > at > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > at > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > ... 12 more > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > org.elasticsearch.transport.RemoteTransportException: > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][1] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > org.elasticsearch.transport.RemoteTransportException: > [esn-02][inet[/10.8.2.47:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][4] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > org.elasticsearch.transport.RemoteTransportException: > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][2] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:56:39,288][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], > [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > heap space]]] > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], > [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > heap space]]] > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], > [R], s[STARTED], reason [master > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked > shard as started, but shard have not been created, mark shard as failed] > [2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > org.elasticsearch.transport.RemoteTransportException: > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][1] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > org.elasticsearch.transport.RemoteTransportException: > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][2] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > org.elasticsearch.transport.RemoteTransportException: > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][4] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:20,317][WARN][cluster.action.shard] [esn-03] > received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], > [R], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > heap space]]] > [2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > org.elasticsearch.transport.RemoteTransportException: > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][2] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > org.elasticsearch.transport.RemoteTransportException: > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][1] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > org.elasticsearch.transport.RemoteTransportException: > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][4] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > org.elasticsearch.transport.RemoteTransportException: > [esn-05][inet[/10.8.2.50:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][0] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > org.elasticsearch.transport.RemoteTransportException: > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][4] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > execute > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > org.elasticsearch.transport.RemoteTransportException: > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > Caused by: org.elasticsearch.index.IndexShardMissingException: > [messages2][1] missing > at > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > at > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen wrote: > > > Hi Drew, > > > > I think using TTL is fine. This could also have happened if a normal > > large delete was triggered. Can you share your hardware details? Also > > would be great if we can see some stacktraces from the log. Can you > > also share your index segments with the indices segment api > > ([http://localhost:9200/\_segments](http://localhost:9200/_segments))? > > > > The index consists out of segments, which are read only (expects for > > deletes). Normally segments get added to the index when data is added > > or updated. When documents get deleted they initially get marked as > > deleted. After some time these segments get merged into a bigger > > segment and then deleted documents are purged. Merging is managed by a > > MergePolicy. Merging is an expensive operation. Luckily a lot can be > > configured regarding to merging. > > > > ES supports throttling for merging, but this isn't enabled by default. > > Enabling this allows you to use less system resources for merging. > > Merges are then slower but then at least don't bring the system down: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > The MergePolicy can also be configured. Sometimes it makes sense to > > not merge large segments. Maybe this also applies in your case. The > > options for merging can be found: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge.html) > > > > Before changing it is wise to check the output of the indices segment > > api. > > > > Martijn > > > > On 15 October 2012 22:34, Drew Daugherty [djdaugh...@gmail.com](mailto:djdaugh...@gmail.com) wrote: > > > > > Hi, > > > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > > documents > > > in it. The index has 6 shards. Disk size consumed by this index is > > > around > > > 6.5 TB (excluding replicas). We set a 30 day default TTL on the index > > > mapping when it was created. On the day the TTL was set to expire, > > > there > > > was sudden large spike in load and swap activity that seems to have > > > brought > > > down nodes in the cluster. The logs contained OOMEs and unfortunately > > > one > > > of our shards became corrupt and will no longer load > > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use > > > TTL > > > settings with very large indexes? We are now considering going to a > > > scheme > > > where we roll our indexes and drop older ones rather than use TTLs. > > > > > > -drew > > > > > > -- > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [October 16, 2012, 10:24am UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/5 "2012-10-16T10:24:26Z")
From the stacktraces it seems that the OOM occurs after the merging has completed and during the new view on the index is being opened. Are using sorting or faceting? This takes usually a lot of memory. Can you share you nodes info ([http://localhost:9200/\_nodes/stats?all](http://localhost:9200/_nodes/stats?all))? Perhaps you need to increase the number of nodes or increase the heap space size. Martijn On 16 October 2012 01:37, Drew Daugherty [djdaugherty612@gmail.com](mailto:djdaugherty612@gmail.com) wrote: > Sorry I meant 12GB for heap memory. > > -drew > > On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote: > > > Thanks Martijn, > > > > I have attached the output from the segments command. Hardware consists of > > 6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of heap > > memory to the ES JVM. Log entries from the master are below. > > > > -drew > > > > [2012-10-13 00:09:21,820][INFO][monitor.jvm] [esn-03] > > [gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], total > > [728ms]/[53.4m], memory [10.5gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code Cache] > > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] > > [211.6mb]-\>[36.7mb]/[216.3mb]}{[Par Survivor Space] > > [26.4mb]-\>[27mb]/[27mb]}{[CMS Old Gen] [10.2gb]-\>[10.3gb]/[11.7gb]}{[CMS > > Perm Gen] [47mb]-\>[47mb]/[82mb]} > > [2012-10-13 04:21:59,041][WARN][monitor.jvm] [esn-03] > > [gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], total > > [2.5s]/[54.3m], memory [10.4gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code Cache] > > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] [38.5mb]-\>[2.6mb]/[216.3mb]}{[Par > > Survivor Space] [26.7mb]-\>[26.9mb]/[27mb]}{[CMS Old Gen] > > [10.3gb]-\>[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]-\>[47mb]/[82mb]} > > [2012-10-13 09:51:00,359][WARN][transport] [esn-03] > > Received response for a request that has timed out, sent [163943ms] ago, > > timed out [133943ms] ago, action [discovery/zen/fd/ping], node > > [[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id > > [9346581] > > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > space]]] > > [2012-10-13 09:55:58,460][WARN][index.engine.robin] [esn-03] > > [messages2][0] failed engine > > java.lang.OutOfMemoryError: Java heap space > > at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > > at > > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > > at > > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > > at > > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > > at > > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > > at > > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > at > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > at > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > > at > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > > at > > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > > at > > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > space]]] > > [2012-10-13 09:56:02,853][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], > > s[STARTED], reason [master > > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked > > shard as started, but shard have not been created, mark shard as failed] > > [2012-10-13 09:56:03,035][WARN][index.engine.robin] [esn-03] > > [messages2][0] failed to flush after setting shard to inactive > > org.elasticsearch.index.engine.FlushFailedEngineException: [messages2][0] > > Flush failed > > at > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > > at > > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > > at > > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.OutOfMemoryError: Java heap space > > at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > > at > > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > > at > > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > > at > > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > > at > > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > > at > > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > at > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > at > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > > at > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > > ... 12 more > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][1] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-02][inet[/10.8.2.47:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][4] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][2] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:56:39,288][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > space]]] > > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > space]]] > > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], > > s[STARTED], reason [master > > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} marked > > shard as started, but shard have not been created, mark shard as failed] > > [2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][1] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][2] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][4] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:20,317][WARN][cluster.action.shard] [esn-03] > > received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > space]]] > > [2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][2] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][1] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][4] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-05][inet[/10.8.2.50:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][0] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][4] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed to > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > org.elasticsearch.transport.RemoteTransportException: > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > [messages2][1] missing > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > at > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > at > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > > > On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen wrote: > > > > > Hi Drew, > > > > > > I think using TTL is fine. This could also have happened if a normal > > > large delete was triggered. Can you share your hardware details? Also > > > would be great if we can see some stacktraces from the log. Can you > > > also share your index segments with the indices segment api > > > ([http://localhost:9200/\_segments](http://localhost:9200/_segments))? > > > > > > The index consists out of segments, which are read only (expects for > > > deletes). Normally segments get added to the index when data is added > > > or updated. When documents get deleted they initially get marked as > > > deleted. After some time these segments get merged into a bigger > > > segment and then deleted documents are purged. Merging is managed by a > > > MergePolicy. Merging is an expensive operation. Luckily a lot can be > > > configured regarding to merging. > > > > > > ES supports throttling for merging, but this isn't enabled by default. > > > Enabling this allows you to use less system resources for merging. > > > Merges are then slower but then at least don't bring the system down: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > The MergePolicy can also be configured. Sometimes it makes sense to > > > not merge large segments. Maybe this also applies in your case. The > > > options for merging can be found: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge.html) > > > > > > Before changing it is wise to check the output of the indices segment > > > api. > > > > > > Martijn > > > > > > On 15 October 2012 22:34, Drew Daugherty [djdaugh...@gmail.com](mailto:djdaugh...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > > > documents > > > > in it. The index has 6 shards. Disk size consumed by this index is > > > > around > > > > 6.5 TB (excluding replicas). We set a 30 day default TTL on the index > > > > mapping when it was created. On the day the TTL was set to expire, > > > > there > > > > was sudden large spike in load and swap activity that seems to have > > > > brought > > > > down nodes in the cluster. The logs contained OOMEs and unfortunately > > > > one > > > > of our shards became corrupt and will no longer load > > > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use > > > > TTL > > > > settings with very large indexes? We are now considering going to a > > > > scheme > > > > where we roll our indexes and drop older ones rather than use TTLs. > > > > > > > > -drew > > > > > > > > -- > > > > > > -- > > > Met vriendelijke groet, > > > > > > Martijn van Groningen > > -- -- Met vriendelijke groet, Martijn van Groningen -- ---
**Author:** ![Drew\_Daugherty](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@Drew\_Daugherty](https://discuss.elastic.co/u/Drew_Daugherty)\ **Post date:** [October 16, 2012, 3:18pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/6 "2012-10-16T15:18:45Z")
See attached for stats all output after a fresh cluster restart (it locked up again last night - OOMEs). We are currently using different settings to cut down the size of our indexes. -drew On Tuesday, October 16, 2012 4:24:51 AM UTC-6, Martijn v Groningen wrote: > From the stacktraces it seems that the OOM occurs after the merging > has completed and during the new view on the index is being opened. > Are using sorting or faceting? This takes usually a lot of memory. Can > you share you nodes info ([http://localhost:9200/\_nodes/stats?all](http://localhost:9200/_nodes/stats?all))? > > Perhaps you need to increase the number of nodes or increase the heap > space size. > > Martijn > > On 16 October 2012 01:37, Drew Daugherty \<[djdaugh...@gmail.com](mailto:djdaugh...@gmail.com)\\> > wrote: > > > Sorry I meant 12GB for heap memory. > > > > -drew > > > > On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote: > > > > > Thanks Martijn, > > > > > > I have attached the output from the segments command. Hardware consists > > > of > > > 6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of > > > heap > > > memory to the ES JVM. Log entries from the master are below. > > > > > > -drew > > > > > > [2012-10-13 00:09:21,820][INFO][monitor.jvm] [esn-03] > > > [gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], > > > total > > > [728ms]/[53.4m], memory [10.5gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code > > > Cache] > > > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] > > > [211.6mb]-\>[36.7mb]/[216.3mb]}{[Par Survivor Space] > > > [26.4mb]-\>[27mb]/[27mb]}{[CMS Old Gen] > > > [10.2gb]-\>[10.3gb]/[11.7gb]}{[CMS > > > Perm Gen] [47mb]-\>[47mb]/[82mb]} > > > [2012-10-13 04:21:59,041][WARN][monitor.jvm] [esn-03] > > > [gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], > > > total > > > [2.5s]/[54.3m], memory [10.4gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code > > > Cache] > > > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] > > > [38.5mb]-\>[2.6mb]/[216.3mb]}{[Par > > > Survivor Space] [26.7mb]-\>[26.9mb]/[27mb]}{[CMS Old Gen] > > > [10.3gb]-\>[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]-\>[47mb]/[82mb]} > > > [2012-10-13 09:51:00,359][WARN][transport] [esn-03] > > > Received response for a request that has timed out, sent [163943ms] > > > ago, > > > timed out [133943ms] ago, action [discovery/zen/fd/ping], node > > [[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id > > > > [9346581] > > > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], > > > [R], > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > > space]]] > > > [2012-10-13 09:55:58,460][WARN][index.engine.robin] [esn-03] > > > [messages2][0] failed engine > > > java.lang.OutOfMemoryError: Java heap space > > > at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > > > at > > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > > > ``` > > > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > > > at > > > > > > ``` > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > > > > ``` > > > at > > > > > > ``` > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > at > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > ``` > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > at > > > > > > ``` > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], > > > [R], > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > > space]]] > > > [2012-10-13 09:56:02,853][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], > > > [R], > > > s[STARTED], reason [master > > > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} > > > marked > > > shard as started, but shard have not been created, mark shard as > > > failed] > > > [2012-10-13 09:56:03,035][WARN][index.engine.robin] [esn-03] > > > [messages2][0] failed to flush after setting shard to inactive > > > org.elasticsearch.index.engine.FlushFailedEngineException: > > > [messages2][0] > > > Flush failed > > > at > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > > > > ``` > > > at > > > > > > ``` > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > at > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > ``` > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > at > > > > > > ``` > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > Caused by: java.lang.OutOfMemoryError: Java heap space > > > at org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > > > at > > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > > > ``` > > > at > > > > > > ``` > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > > > ``` > > > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > > > at > > > > > > ``` > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > > > > ``` > > > ... 12 more > > > > > > ``` > > > > > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][1] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-02][inet[/10.8.2.47:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][4] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][2] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:56:39,288][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], > > > [P], > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > > space]]] > > > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], > > > [R], > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > > space]]] > > > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], > > > [R], > > > s[STARTED], reason [master > > > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} > > > marked > > > shard as started, but shard have not been created, mark shard as > > > failed] > > > [2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] > > > missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][1] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][2] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][4] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:20,317][WARN][cluster.action.shard] [esn-03] > > > received shard failed for [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], > > > [R], > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap > > > space]]] > > > [2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] > > > missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][2] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][1] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][4] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-05][inet[/10.8.2.50:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][0] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] > > > missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][4] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > to > > > execute > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > > > org.elasticsearch.transport.RemoteTransportException: > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > [messages2][1] missing > > > at > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > ``` > > > at > > > > > > ``` > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ``` > > > at > > > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > ``` > > > at java.lang.Thread.run(Thread.java:662) > > > > > > ``` > > > > > > On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen > > > wrote: > > > > > > > Hi Drew, > > > > > > > > I think using TTL is fine. This could also have happened if a normal > > > > large delete was triggered. Can you share your hardware details? Also > > > > would be great if we can see some stacktraces from the log. Can you > > > > also share your index segments with the indices segment api > > > > ([http://localhost:9200/\_segments](http://localhost:9200/_segments))? > > > > > > > > The index consists out of segments, which are read only (expects for > > > > deletes). Normally segments get added to the index when data is added > > > > or updated. When documents get deleted they initially get marked as > > > > deleted. After some time these segments get merged into a bigger > > > > segment and then deleted documents are purged. Merging is managed by a > > > > MergePolicy. Merging is an expensive operation. Luckily a lot can be > > > > configured regarding to merging. > > > > > > > > ES supports throttling for merging, but this isn't enabled by default. > > > > Enabling this allows you to use less system resources for merging. > > > > Merges are then slower but then at least don't bring the system down: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > > > The MergePolicy can also be configured. Sometimes it makes sense to > > > > not merge large segments. Maybe this also applies in your case. The > > > > options for merging can be found: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge.html) > > > > > > > > Before changing it is wise to check the output of the indices segment > > > > api. > > > > > > > > Martijn > > > > > > > > On 15 October 2012 22:34, Drew Daugherty [djdaugh...@gmail.com](mailto:djdaugh...@gmail.com) > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > > > > documents > > > > > in it. The index has 6 shards. Disk size consumed by this index is > > > > > around > > > > > 6.5 TB (excluding replicas). We set a 30 day default TTL on the > > > > > index > > > > > mapping when it was created. On the day the TTL was set to expire, > > > > > there > > > > > was sudden large spike in load and swap activity that seems to have > > > > > brought > > > > > down nodes in the cluster. The logs contained OOMEs and > > > > > unfortunately > > > > > one > > > > > of our shards became corrupt and will no longer load > > > > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to > > > > > use > > > > > TTL > > > > > settings with very large indexes? We are now considering going to a > > > > > scheme > > > > > where we roll our indexes and drop older ones rather than use TTLs. > > > > > > > > > > -drew > > > > > > > > > > -- > > > > > > > > -- > > > > Met vriendelijke groet, > > > > > > > > Martijn van Groningen > > > > -- > > -- > Met vriendelijke groet, > > Martijn van Groningen -- ---
**Author:** ![Drew\_Daugherty](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@Drew\_Daugherty](https://discuss.elastic.co/u/Drew_Daugherty)\ **Post date:** [October 16, 2012, 4:54pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/7 "2012-10-16T16:54:09Z")
Martijn, See attached for all stats after things are up and indexing documents. -drew On Tuesday, October 16, 2012 9:18:46 AM UTC-6, Drew Daugherty wrote: > See attached for stats all output after a fresh cluster restart (it locked > up again last night - OOMEs). We are currently using different settings to > cut down the size of our indexes. > > -drew > > On Tuesday, October 16, 2012 4:24:51 AM UTC-6, Martijn v Groningen wrote: > > > From the stacktraces it seems that the OOM occurs after the merging > > has completed and during the new view on the index is being opened. > > Are using sorting or faceting? This takes usually a lot of memory. Can > > you share you nodes info ([http://localhost:9200/\_nodes/stats?all](http://localhost:9200/_nodes/stats?all))? > > > > Perhaps you need to increase the number of nodes or increase the heap > > space size. > > > > Martijn > > > > On 16 October 2012 01:37, Drew Daugherty [djdaugh...@gmail.com](mailto:djdaugh...@gmail.com) wrote: > > > > > Sorry I meant 12GB for heap memory. > > > > > > -drew > > > > > > On Monday, October 15, 2012 5:35:32 PM UTC-6, Drew Daugherty wrote: > > > > > > > Thanks Martijn, > > > > > > > > I have attached the output from the segments command. Hardware > > > > consists of > > > > 6 nodes, each with 16 cores and 24GB of memory. I allocated 24GB of > > > > heap > > > > memory to the ES JVM. Log entries from the master are below. > > > > > > > > -drew > > > > > > > > [2012-10-13 00:09:21,820][INFO][monitor.jvm] [esn-03] > > > > [gc][ParNew][696037][116970] duration [728ms], collections [1]/[1s], > > > > total > > > > [728ms]/[53.4m], memory [10.5gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code > > > > Cache] > > > > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] > > > > [211.6mb]-\>[36.7mb]/[216.3mb]}{[Par Survivor Space] > > > > [26.4mb]-\>[27mb]/[27mb]}{[CMS Old Gen] > > > > [10.2gb]-\>[10.3gb]/[11.7gb]}{[CMS > > > > Perm Gen] [47mb]-\>[47mb]/[82mb]} > > > > [2012-10-13 04:21:59,041][WARN][monitor.jvm] [esn-03] > > > > [gc][ParNew][711168][118683] duration [2.5s], collections [1]/[3.4s], > > > > total > > > > [2.5s]/[54.3m], memory [10.4gb]-\>[10.4gb]/[11.9gb], all\_pools {[Code > > > > Cache] > > > > [9.5mb]-\>[9.5mb]/[48mb]}{[Par Eden Space] > > > > [38.5mb]-\>[2.6mb]/[216.3mb]}{[Par > > > > Survivor Space] [26.7mb]-\>[26.9mb]/[27mb]}{[CMS Old Gen] > > > > [10.3gb]-\>[10.4gb]/[11.7gb]}{[CMS Perm Gen] [47mb]-\>[47mb]/[82mb]} > > > > [2012-10-13 09:51:00,359][WARN][transport] [esn-03] > > > > Received response for a request that has timed out, sent [163943ms] > > > > ago, > > > > timed out [133943ms] ago, action [discovery/zen/fd/ping], node > > > > [[esn-02][q6Al5C7HQ-a2TJotgg1ojg][inet[/10.8.2.47:9300]]{master=false}], id > > > > > > [9346581] > > > > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][4], > > > > node[q6Al5C7HQ-a2TJotgg1ojg], [R], > > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > > > > heap > > > > space]]] > > > > [2012-10-13 09:55:58,460][WARN][index.engine.robin] [esn-03] > > > > [messages2][0] failed engine > > > > java.lang.OutOfMemoryError: Java heap space > > > > at > > > > org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > > > > at > > > > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > > > > > ``` > > > > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > > > > at > > > > > > > > ``` > > > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:55:58,453][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][1], > > > > node[5IAi9fteSt6OGHyXH7VrTw], [R], > > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > > > > heap > > > > space]]] > > > > [2012-10-13 09:56:02,853][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][1], > > > > node[5IAi9fteSt6OGHyXH7VrTw], [R], > > > > s[STARTED], reason [master > > > > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} > > > > marked > > > > shard as started, but shard have not been created, mark shard as > > > > failed] > > > > [2012-10-13 09:56:03,035][WARN][index.engine.robin] [esn-03] > > > > [messages2][0] failed to flush after setting shard to inactive > > > > org.elasticsearch.index.engine.FlushFailedEngineException: > > > > [messages2][0] > > > > Flush failed > > > > at > > > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:837) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.index.engine.robin.RobinEngine.updateIndexingBufferSize(RobinEngine.java:219) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.indices.memory.IndexingMemoryController$ShardsIndicesStatusChecker.run(IndexingMemoryController.java:178) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.threadpool.ThreadPool$LoggingRunnable.run(ThreadPool.java:288) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > Caused by: java.lang.OutOfMemoryError: Java heap space > > > > at > > > > org.apache.lucene.util.packed.Packed64.(Packed64.java:122) > > > > at > > > > org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:221) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:81) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.TermInfosReaderIndex.(TermInfosReaderIndex.java:84) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.TermInfosReader.(TermInfosReader.java:116) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:83) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > > > > > ``` > > > > at org.apache.lucene.index.IndexReader.open(IndexReader.java:296) > > > > at > > > > > > > > ``` > > > > org.apache.lucene.search.SearcherManager.(SearcherManager.java:82) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1364) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:831) > > > > > > ``` > > > > ... 12 more > > > > > > > > ``` > > > > > > > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][1] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][4], node[q6Al5C7HQ-a2TJotgg1ojg], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-02][inet[/10.8.2.47:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][4] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:56:05,558][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@51733937] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][2] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:56:39,288][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][4], > > > > node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], > > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > > > > heap > > > > space]]] > > > > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][2], > > > > node[hLtNHFLNRjStSNdqzk4HNg], [R], > > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > > > > heap > > > > space]]] > > > > [2012-10-13 09:57:34,414][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][2], > > > > node[hLtNHFLNRjStSNdqzk4HNg], [R], > > > > s[STARTED], reason [master > > > > [esn-03][6Axy06o6SbqqdoeiQdxCBw][inet[/10.8.2.48:9300]]{master=true} > > > > marked > > > > shard as started, but shard have not been created, mark shard as > > > > failed] > > > > [2012-10-13 09:58:19,785][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] > > > > missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:19,796][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][1] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:19,801][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][2] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:19,802][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@46772584] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][4] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:20,317][WARN][cluster.action.shard] [esn-03] > > > > received shard failed for [messages2][0], > > > > node[BD4mbA1lSYezrejzi-J-fA], [R], > > > > s[STARTED], reason [engine failure, message [OutOfMemoryError[Java > > > > heap > > > > space]]] > > > > [2012-10-13 09:58:28,655][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] > > > > missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,703][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][2], node[hLtNHFLNRjStSNdqzk4HNg], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-04][inet[/10.8.2.49:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][2] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][1] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][4] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,704][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][0], node[BD4mbA1lSYezrejzi-J-fA], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@430234c5] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-05][inet[/10.8.2.50:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][0] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,735][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][0], node[6Axy06o6SbqqdoeiQdxCBw], [P], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > > > > > org.elasticsearch.index.IndexShardMissingException: [messages2][0] > > > > missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:234) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:211) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:187) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][4], node[Cgz1EWFfS1qM5c4SLa-dyQ], [P], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-01][inet[/10.8.2.46:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][4] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > [2012-10-13 09:58:28,738][DEBUG][action.admin.indices.stats] [esn-03] > > > > [messages2][1], node[5IAi9fteSt6OGHyXH7VrTw], [R], s[STARTED]: Failed > > > > to > > > > execute > > > > [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@f861dc6] > > > > > > org.elasticsearch.transport.RemoteTransportException: > > > > [esn-00][inet[/10.8.2.45:9300]][indices/stats/s] > > > > Caused by: org.elasticsearch.index.IndexShardMissingException: > > > > [messages2][1] missing > > > > at > > > > org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:179) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:145) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:53) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:398) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$ShardTransportHandler.messageReceived(TransportBroadcastOperationAction.java:384) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:400) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > ``` > > > > at > > > > > > > > ``` > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > ``` > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > ``` > > > > > > > > On Monday, October 15, 2012 3:46:49 PM UTC-6, Martijn v Groningen > > > > wrote: > > > > > > > > > Hi Drew, > > > > > > > > > > I think using TTL is fine. This could also have happened if a normal > > > > > large delete was triggered. Can you share your hardware details? Also > > > > > would be great if we can see some stacktraces from the log. Can you > > > > > also share your index segments with the indices segment api > > > > > ([http://localhost:9200/\_segments](http://localhost:9200/_segments))? > > > > > > > > > > The index consists out of segments, which are read only (expects for > > > > > deletes). Normally segments get added to the index when data is added > > > > > or updated. When documents get deleted they initially get marked as > > > > > deleted. After some time these segments get merged into a bigger > > > > > segment and then deleted documents are purged. Merging is managed by > > > > > a > > > > > MergePolicy. Merging is an expensive operation. Luckily a lot can be > > > > > configured regarding to merging. > > > > > > > > > > ES supports throttling for merging, but this isn't enabled by > > > > > default. > > > > > Enabling this allows you to use less system resources for merging. > > > > > Merges are then slower but then at least don't bring the system down: > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > > > > > The MergePolicy can also be configured. Sometimes it makes sense to > > > > > not merge large segments. Maybe this also applies in your case. The > > > > > options for merging can be found: > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge.html) > > > > > > > > > > Before changing it is wise to check the output of the indices segment > > > > > api. > > > > > > > > > > Martijn > > > > > > > > > > On 15 October 2012 22:34, Drew Daugherty [djdaugh...@gmail.com](mailto:djdaugh...@gmail.com) > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > > > > > documents > > > > > > in it. The index has 6 shards. Disk size consumed by this index > > > > > > is > > > > > > around > > > > > > 6.5 TB (excluding replicas). We set a 30 day default TTL on the > > > > > > index > > > > > > mapping when it was created. On the day the TTL was set to expire, > > > > > > there > > > > > > was sudden large spike in load and swap activity that seems to have > > > > > > brought > > > > > > down nodes in the cluster. The logs contained OOMEs and > > > > > > unfortunately > > > > > > one > > > > > > of our shards became corrupt and will no longer load > > > > > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to > > > > > > use > > > > > > TTL > > > > > > settings with very large indexes? We are now considering going to > > > > > > a > > > > > > scheme > > > > > > where we roll our indexes and drop older ones rather than use TTLs. > > > > > > > > > > > > -drew > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > Met vriendelijke groet, > > > > > > > > > > Martijn van Groningen > > > > > > -- > > > > -- > > Met vriendelijke groet, > > > > Martijn van Groningen -- ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [October 16, 2012, 5:11pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/8 "2012-10-16T17:11:12Z")
I haven't used TTL before, so am not of much help there, but using a rolling index approach will definitely be much more efficient. Deletes bloat your index size until segments are merged and they're expunged. With rolling indexes you won't have the same bloat. On Monday, October 15, 2012 2:34:39 PM UTC-6, Drew Daugherty wrote: > Hi, > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > documents in it. The index has 6 shards. Disk size consumed by this index > is around 6.5 TB (excluding replicas). We set a 30 day default TTL on the > index mapping when it was created. On the day the TTL was set to expire, > there was sudden large spike in load and swap activity that seems to have > brought down nodes in the cluster. The logs contained OOMEs and > unfortunately one of our shards became corrupt and will no longer load > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL > settings with very large indexes? We are now considering going to a scheme > where we roll our indexes and drop older ones rather than use TTLs. > > -drew -- ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [October 16, 2012, 10:08pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/9 "2012-10-16T22:08:59Z")
Drew, What kind of queries / filter executing in general? Are you using the \_parent type in your mapping? Just wondering what else might take that amount of memory. Perhaps you can lower the amount segments that are merged at a time. This can be done by configuring the 'index.merge.policy.max\_merge\_at\_once' option in either the elasticsearch.yml on each node or via the update indices settings api. Merging segments that contain deletes takes up more ram than segments without deletes. In your index most segment do contain deletes. The default 'index.merge.policy.max\_merge\_at\_once' is 10, Maybe you can try with a value of 5. Like ppearcy says, for time based data the rolling index approach doesn't have the merging overhead. If that is possible for you that would be a better solution. Martijn On 16 October 2012 19:11, ppearcy [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > I haven't used TTL before, so am not of much help there, but using a rolling > index approach will definitely be much more efficient. Deletes bloat your > index size until segments are merged and they're expunged. With rolling > indexes you won't have the same bloat. > > On Monday, October 15, 2012 2:34:39 PM UTC-6, Drew Daugherty wrote: > > > Hi, > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > documents in it. The index has 6 shards. Disk size consumed by this index > > is around 6.5 TB (excluding replicas). We set a 30 day default TTL on the > > index mapping when it was created. On the day the TTL was set to expire, > > there was sudden large spike in load and swap activity that seems to have > > brought down nodes in the cluster. The logs contained OOMEs and > > unfortunately one of our shards became corrupt and will no longer load > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use TTL > > settings with very large indexes? We are now considering going to a scheme > > where we roll our indexes and drop older ones rather than use TTLs. > > > > -drew > > -- -- Met vriendelijke groet, Martijn van Groningen -- ---
**Author:** ![Drew\_Daugherty](https://avatars.discourse-cdn.com/v4/letter/d/6bbea6/32.png) [@Drew\_Daugherty](https://discuss.elastic.co/u/Drew_Daugherty)\ **Post date:** [October 17, 2012, 3:30pm UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/10 "2012-10-17T15:30:36Z")
So far I don't think we have had a query load that would impact memory usage significantly. We are still at the stage of ad hoc querying where one or two people might be testing things. There has been interest in using faceting but I haven't yet confirmed it is actually being used. We are not using \_parent type in our mappings. Thanks for the information about tweaking merge settings. I think we will try rolling indexes to avoid the overhead. -drew On Tuesday, October 16, 2012 4:09:23 PM UTC-6, Martijn v Groningen wrote: > Drew, > > What kind of queries / filter executing in general? Are you using the > \_parent type in your mapping? Just wondering what else might take that > amount of memory. > > Perhaps you can lower the amount segments that are merged at a time. > This can be done by configuring the > 'index.merge.policy.max\_merge\_at\_once' option in either the > elasticsearch.yml on each node or via the update indices settings api. > Merging segments that contain deletes takes up more ram than segments > without deletes. In your index most segment do contain deletes. The > default 'index.merge.policy.max\_merge\_at\_once' is 10, Maybe you can > try with a value of 5. > > Like ppearcy says, for time based data the rolling index approach > doesn't have the merging overhead. > If that is possible for you that would be a better solution. > > Martijn > > On 16 October 2012 19:11, ppearcy \<[ppe...@gmail.com](mailto:ppe...@gmail.com) \\> wrote: > > > I haven't used TTL before, so am not of much help there, but using a > > rolling > > index approach will definitely be much more efficient. Deletes bloat > > your > > index size until segments are merged and they're expunged. With rolling > > indexes you won't have the same bloat. > > > > On Monday, October 15, 2012 2:34:39 PM UTC-6, Drew Daugherty wrote: > > > > > Hi, > > > > > > I am using elasticsearch 0.19.8 and have a large index with 300M+ > > > documents in it. The index has 6 shards. Disk size consumed by this > > > index > > > is around 6.5 TB (excluding replicas). We set a 30 day default TTL on > > > the > > > index mapping when it was created. On the day the TTL was set to > > > expire, > > > there was sudden large spike in load and swap activity that seems to > > > have > > > brought down nodes in the cluster. The logs contained OOMEs and > > > unfortunately one of our shards became corrupt and will no longer load > > > (IndexShardMissingExceptions). Is this a bug? Is it advisable to use > > > TTL > > > settings with very large indexes? We are now considering going to a > > > scheme > > > where we roll our indexes and drop older ones rather than use TTLs. > > > > > > -drew > > > > -- > > -- > Met vriendelijke groet, > > Martijn van Groningen -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:08am UTC](https://discuss.elastic.co/t/ttl-load-problems/9361/11 "2017-07-06T03:08:17Z")