EDIT: I opened a thread also on SO, where I've updated with more information my use case: http://stackoverflow.com/questions/23418490/elasticsearch-high-number-of-index-causes-oom
Hi all,
I've been struggling with this problem for quite a bit and now I decided to post a question on the topic to seek for some advice.
I'm testing the environment for indexing a big number of data over time. Basically everyday I'll index logs and associated documents from various websites.
I want to have one index per website to have a better logical division and to be able to filter the queries and have faster responses.
I foresee a traffic of ~1Gb of data per day per index.
I'm testing the environment on an AWS instance with 2x80Gb SSDs, CPU vendor: Intel, CPU model: Xeon (2500 MHz), CPU total logical cores: 8
I have 1 node, x indexes and 1 shard per index (for the test environment) and pushing documents taken from wikipedia articles, randomly distributing them on 200 indexes or more.
I'm indexing documents form a Queue, with a single client at the rate of about 200/s. This is just for testing. When I'll be able to fix the OOM issue, I'll increase the number of clients and the number of nodes.
I have I tried with 500k docs and 200 indexes and it is fine, if I try with 500k and 300 indexes, it throws the hated OOM error exception. Also 1m documents and 100 indexes throws back an OOM error.
I'm running now tests with 1m documents and 200 indexes.
I tried changing the shard number, and I also tried to play with these parameters
index.merge.policy.max_merged_segment: 2g
index.merge.policy.segments_per_tier: [ I tried 5 (and reduced the max number too) - 25]
index.merge.policy.max_merge_at_once: 8
My settings are
{
"cluster_name" : "elasticsearch",
"nodes" : {
"QBk8YzISQPu-VVMnKvhEmQ" : {
"name" : "",
"transport_address" : "",
"host" : "",
"ip" : "",
"version" : "1.1.0",
"build" : "2181e11",
"http_address" : "",
"settings" : {
"path" : {
"data" : "/mnt/db/se_data/elasticsearch/",
"logs" : "/mnt/db/searchengines/elasticsearch-1.1.0/logs",
"home" : "/mnt/db/searchengines/elasticsearch-1.1.0"
},
"cluster" : {
"name" : "elasticsearch"
},
"index" : {
"number_of_shards" : "1"
},
"foreground" : "yes",
"name" : "",
"max-open-files" : "true"
},
"os" : {
"refresh_interval" : 1000,
"available_processors" : 8,
"cpu" : {
"vendor" : "Intel",
"model" : "Xeon",
"mhz" : 2500,
"total_cores" : 8,
"total_sockets" : 8,
"cores_per_socket" : 32,
"cache_size_in_bytes" : 25600
},
"mem" : {
"total_in_bytes" : 31502180352
},
"swap" : {
"total_in_bytes" : 3071995904
}
},
"process" : {
"refresh_interval" : 1000,
"id" : 20155,
"max_file_descriptors" : 64000,
"mlockall" : false
},
"jvm" : {
"pid" : 20155,
"version" : "1.7.0_51",
"vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
"vm_version" : "24.51-b03",
"vm_vendor" : "Oracle Corporation",
"start_time" : 1398962098305,
"mem" : {
"heap_init_in_bytes" : 268435456,
"heap_max_in_bytes" : 10667687936,
"non_heap_init_in_bytes" : 24313856,
"non_heap_max_in_bytes" : 136314880,
"direct_max_in_bytes" : 10667687936
},
"gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],
"memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]
}
}
}
}
Here are the JVM settings:
-Xms256m -Xmx10g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Delasticsearch -Des.foreground=yes
max numbers of file open is set to 65k (this was the previous issue i had.. the number of allowed open files was only 4k)
Checking the memory consumption with bigdesk, I see the Heap Memory committed/used grows constantly while reaching the crash-point. I think that is causing the OOM error even though it crashes when the consumption reaches ~4-6Gb towards a max allocated java heap of 10Gb.
Could it be a GC problem? Should I follow the settings from this article to tune it?
If I insert a flush() command every, say, 10k inserted documents on any index, could it help reducing the memory usage?
Is the number of indexes too high? Should I change the approach? Put more logs coming from different websites in a single index seem to be the most plausible solution, is it?
PS: I'm using and a python script to test the environment, with the elasticsearch for python module (should I use pyelasticsearch instead? It should be just a feature-richer module, right?).
Let me know if you need more info.
Thank you for your time!
The stacktrace from the ES log is the following:
[2014-05-01 00:16:49,526][INFO ][node ] [Captain Wings] stopping ...
[2014-05-01 00:16:49,767][WARN ][index.shard.service ] [Captain Wings] [index_220][0] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException: [index_220][0] Refresh failed
at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:725)
at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:469)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:920)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: _75.fdt
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.close(CompressingStoredFieldsWriter.java:138)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.index.StoredFieldsProcessor.flush(StoredFieldsProcessor.java:78)
at org.apache.lucene.index.TwoStoredFieldsConsumers.flush(TwoStoredFieldsConsumers.java:41)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:80)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:465)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:506)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:616)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:370)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:288)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:253)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:118)
at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:706)
... 5 more
Suppressed: java.io.FileNotFoundException: _75.fdx
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter.close(CompressingStoredFieldsIndexWriter.java:205)
... 24 more
[2014-05-01 00:16:49,804][WARN ][index.merge.scheduler ] [Captain Wings] [index_65][0] failed to merge
java.io.FileNotFoundException: _7n_es090_0.tim
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81)
at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371)
at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:163)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Suppressed: java.io.FileNotFoundException: _7n_es090_0.tip
... 26 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.doc
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587)
... 23 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.pos
... 28 more
[2014-05-01 00:16:49,807][WARN ][index.engine.internal ] [Captain Wings] [index_65][0] failed engine
org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: _7n_es090_0.tim
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:92)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: _7n_es090_0.tim
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81)
at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371)
at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:163)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
... 9 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.tip
... 26 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.doc
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587)
... 23 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.pos
... 28 more
[2014-05-01 00:16:49,807][WARN ][index.shard.service ] [Captain Wings] [index_65][0] Failed to perform scheduled engine optimize/merge
org.elasticsearch.index.engine.OptimizeFailedEngineException: [index_65][0] Optimize failed
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:936)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: _7n_es090_0.tim
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:93)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
... 4 more
Caused by: java.io.FileNotFoundException: _7n_es090_0.tim
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81)
at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371)
at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:163)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
... 9 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.tip
... 26 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.doc
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587)
... 23 more
Suppressed: java.io.FileNotFoundException: _7n_es090_0.pos
... 28 more
[2014-05-01 00:16:49,913][WARN ][cluster.action.shard ] [Captain Wings] [index_65][0] sending failed shard for [index_65][0], node[Fj243PXdSNGtSm_jLNl9hQ], [P], s[STARTED], indexUUID [v6ZQJKpTRbiZZ5e6Q6ENyw], reason [engine failure, message [MergeException[java.io.FileNotFoundException: _7n_es090_0.tim]; nested: FileNotFoundException[_7n_es090_0.tim]; ]]
[2014-05-01 00:16:49,913][WARN ][cluster.action.shard ] [Captain Wings] [index_65][0] received shard failed for [index_65][0], node[Fj243PXdSNGtSm_jLNl9hQ], [P], s[STARTED], indexUUID [v6ZQJKpTRbiZZ5e6Q6ENyw], reason [engine failure, message [MergeException[java.io.FileNotFoundException: _7n_es090_0.tim]; nested: FileNotFoundException[_7n_es090_0.tim]; ]]
[2014-05-01 00:17:03,453][WARN ][index.merge.scheduler ] [Captain Wings] [index_221][0] failed to merge
java.io.FileNotFoundException: _7e_es090_0.tim
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81)
at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371)
at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:163)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Suppressed: java.io.FileNotFoundException: _7e_es090_0.tip
... 26 more
Suppressed: java.io.FileNotFoundException: _7e_es090_0.doc
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587)
... 23 more
Suppressed: java.io.FileNotFoundException: _7e_es090_0.pos
... 28 more
[2014-05-01 00:17:03,454][WARN ][index.engine.internal ] [Captain Wings] [index_221][0] failed engine
org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: _7e_es090_0.tim
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:92)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException: _7e_es090_0.tim
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81)
at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371)
at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:163)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
... 9 more
Suppressed: java.io.FileNotFoundException: _7e_es090_0.tip
... 26 more
Suppressed: java.io.FileNotFoundException: _7e_es090_0.doc
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261)
at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:140)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587)
... 23 more
Suppressed: java.io.FileNotFoundException: _7e_es090_0.pos
... 28 more
[2014-05-01 00:17:06,961][INFO ][node ] [Captain Wings] stopped
[2014-05-01 00:17:06,961][INFO ][node ] [Captain Wings] closing ...
[2014-05-01 00:17:06,966][INFO ][node ] [Captain Wings] closed