# Repetitive IndexFailedEngineException crash **URL:** \ **Category:** Elasticsearch\ **Created:** [March 5, 2013, 5:27pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019 "2013-03-05T17:27:30Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Kranan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kranan/32/2455_2.png) [@Kranan](https://discuss.elastic.co/u/Kranan)\ **Post date:** [March 5, 2013, 5:27pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/1 "2013-03-05T17:27:30Z")
I am using Graylog2 w/ ElasticSearch for a year now and it worked perfectly... since last month. After we changed the servers architecture (but not the server itself), I am experiencing repetitive crash of the ElasticSearch service. So I upgraded the Graylog2 to its last version (0.10.0, and of course ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it still crashes every 5-6 hours, sometimes less. And after each crash, ElasticSearch index is gone and starting all over again. Here is an extract of the /var/log/elasticsearch/graylog2.log after the crash : org.elasticsearch.index.engine.IndexFailedEngineException: [graylog2\_recent][0] Index failed for [message#3b25a760-851f-11e2-ab8c-0024e82dd6ba] at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494) at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.FileNotFoundException: /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_recent/0/index/\_16bi.fdt (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) at org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83) at org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64) at org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107) at org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151) at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404) at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424) at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577) at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489) ... 7 more /etc/init.d/elasticsearch memory conf : ES\_MIN\_MEM=256m ES\_MAX\_MEM=4g Graylog2 and elasticsearch is hosted on the same server, no shard, no replica. Hope someone can help 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 5, 2013, 6:45pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/2 "2013-03-05T18:45:46Z")
Your index is corrupted because the max number of open files has been exceeded. Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch' # File /etc/security/limits.conf elasticsearch - nofile 65535 elasticsearch - memlock unlimited Jörg Am 05.03.13 18:27, schrieb Kranan: > I am using Graylog2 w/ Elasticsearch for a year now and it worked > perfectly... since last month. > After we changed the servers architecture (but not the server itself), > I am experiencing repetitive > crash of the Elasticsearch service. > > So I upgraded the Graylog2 to its last version (0.10.0, and of course > Elasticsearch from the 0.19.10 to the 0.20.4 as required), but it > still crashes every 5-6 hours, sometimes less. > And after each crash, Elasticsearch index is gone and starting all > over again. > > Here is an extract of the /var/log/elasticsearch/graylog2.log after > the crash : > > ``` > org.elasticsearch.index.engine.IndexFailedEngineException: > [graylog2_recent][0] Index failed for > [message#3b25a760-851f-11e2-ab8c-0024e82dd6ba] > at > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494) > at > org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.FileNotFoundException: > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/0/index/_16bi.fdt > (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) > at > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) > at > org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) > at > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) > at > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) > at > org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83) > at > org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64) > at > org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107) > at > org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151) > at > org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404) > at > org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424) > at > org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043) > at > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772) > at > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) > at > org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577) > at > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489) > ... 7 more > > ``` > > /etc/init.d/elasticsearch memory conf : > > ``` > ES_MIN_MEM=256m > ES_MAX_MEM=4g > > ``` > > Graylog2 and elasticsearch is hosted on the same server, no shard, no > replica. > > ## Hope someone can help 🙂 > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Kranan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kranan/32/2455_2.png) [@Kranan](https://discuss.elastic.co/u/Kranan)\ **Post date:** [March 6, 2013, 1:56pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/3 "2013-03-06T13:56:59Z")
Thanx. I tried your conf and after a few hours, it freezed the server (ssh) for a while and erased/corrupted the index as well. And /var/log/elasticsearch is empty. On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote: > Your index is corrupted because the max number of open files has been > exceeded. > > Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch' > > # File /etc/security/limits.conf > > elasticsearch - nofile 65535 > elasticsearch - memlock unlimited > > Jörg > > Am 05.03.13 18:27, schrieb Kranan: > > > I am using Graylog2 w/ Elasticsearch for a year now and it worked > > perfectly... since last month. > > After we changed the servers architecture (but not the server itself), > > I am experiencing repetitive > > crash of the Elasticsearch service. > > > > So I upgraded the Graylog2 to its last version (0.10.0, and of course > > Elasticsearch from the 0.19.10 to the 0.20.4 as required), but it > > still crashes every 5-6 hours, sometimes less. > > And after each crash, Elasticsearch index is gone and starting all > > over again. > > > > Here is an extract of the /var/log/elasticsearch/graylog2.log after > > the crash : > > > > ``` > > org.elasticsearch.index.engine.IndexFailedEngineException: > > [graylog2_recent][0] Index failed for > > [message#3b25a760-851f-11e2-ab8c-0024e82dd6ba] > > at > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330) > > > ``` > > at > > > > ``` > > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158) > > > ``` > > at > > > > ``` > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531) > > > ``` > > at > > > > ``` > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429) > > > ``` > > at > > > > ``` > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > ``` > > at > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > ``` > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException: > > > > ``` > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_recent/0/index/\_16bi.fdt > > > ``` > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at > > > > ``` > > java.io.RandomAccessFile.(RandomAccessFile.java:233) > > > ``` > > at > > > > ``` > > org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) > > > ``` > > at > > > > ``` > > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) > > > ``` > > at > > > > ``` > > org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) > > > ``` > > at > > org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83) > > at > > > > ``` > > org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489) > > > ``` > > ... 7 more > > > > ``` > > > > /etc/init.d/elasticsearch memory conf : > > > > ``` > > ES_MIN_MEM=256m > > ES_MAX_MEM=4g > > > > ``` > > > > Graylog2 and elasticsearch is hosted on the same server, no shard, no > > replica. > > > > ## Hope someone can help 🙂 > > > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Kranan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kranan/32/2455_2.png) [@Kranan](https://discuss.elastic.co/u/Kranan)\ **Post date:** [March 6, 2013, 2:02pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/4 "2013-03-06T14:02:18Z")
Here is the log of Elasticsearch booting up : [2013-03-06 14:29:46,484][INFO][node] [Bench, Morris] {0.20.4}[7148]: initializing ... [2013-03-06 14:29:46,511][INFO][plugins] [Bench, Morris] loaded , sites [2013-03-06 14:29:50,383][INFO][node] [Bench, Morris] {0.20.4}[7148]: initialized [2013-03-06 14:29:50,383][INFO][node] [Bench, Morris] {0.20.4}[7148]: starting ... [2013-03-06 14:29:50,545][INFO][transport] [Bench, Morris] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/xxx.xxx.xxx.150:9300]} [2013-03-06 14:29:53,629][INFO][cluster.service] [Bench, Morris] new\_master [Bench, Morris][_-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason: zen-disco-join (elected\_as\_master) [2013-03-06 14:29:53,636][INFO][discovery] [Bench, Morris] graylog2/_-ugh0MlRdymGhCE8gYl-w [2013-03-06 14:29:53,707][INFO][http] [Bench, Morris] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/xxx.xxx.xxx.150:9200]} [2013-03-06 14:29:53,707][INFO][node] [Bench, Morris] {0.20.4}[7148]: started [2013-03-06 14:30:00,244][INFO][gateway] [Bench, Morris] recovered [2] indices into cluster\_state [2013-03-06 14:30:00,267][INFO][cluster.service] [Bench, Morris] added {[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true, data=false, master=false},}, reason: zen-disco-receive(join from node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true, data=false, master=false}]) [2013-03-06 14:30:06,454][WARN][index.shard.service] [Bench, Morris] [graylog2\_0][1] Failed to perform scheduled engine refresh org.elasticsearch.index.engine.RefreshFailedEngineException: [graylog2\_0][1] Refresh failed at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.FileNotFoundException: /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/1/index/\_nwa.nrm (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) ... 5 more [2013-03-06 14:30:06,459][WARN][index.merge.scheduler] [Bench, Morris] [graylog2\_0][1] failed to merge java.io.FileNotFoundException: /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/1/index/\_nwa.nrm (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) [2013-03-06 14:30:06,499][WARN][index.shard.service] [Bench, Morris] [graylog2\_0][2] Failed to perform scheduled engine refresh org.elasticsearch.index.engine.RefreshFailedEngineException: [graylog2\_0][2] Refresh failed at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.FileNotFoundException: /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/2/index/\_o6g.fdx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133) at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) ... 5 more [2013-03-06 14:30:06,792][WARN][index.shard.service] [Bench, Morris] [graylog2\_recent][3] Failed to perform scheduled engine refresh org.elasticsearch.index.engine.RefreshFailedEngineException: [graylog2\_recent][3] Refresh failed at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.FileNotFoundException: /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_recent/3/index/\_m9b.frq (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) at org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) at org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) at org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47) at org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33) at org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51) at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113) at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70) at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60) at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581) at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587) at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) ... 5 more [2013-03-06 14:30:06,917][WARN][indices.cluster] [Bench, Morris] [graylog2\_0][3] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [graylog2\_0][3] failed recovery at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: org.elasticsearch.index.engine.FlushFailedEngineException: [graylog2\_0][3] Flush failed at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) ... 3 more Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException: [graylog2\_0][3] Refresh failed at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) at org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967) at org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892) ... 6 more Caused by: java.io.FileNotFoundException: /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/_nyr.fdt (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:233) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131) at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) at org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) at org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) at org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) ... 8 more [2013-03-06 14:30:07,003][WARN][cluster.action.shard] [Bench, Morris] sending failed shard for [graylog2\_0][3], node[_-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2\_0][3] failed recovery]; nested: FlushFailedEngineException[[graylog2\_0][3] Flush failed]; nested: RefreshFailedEngineException[[graylog2\_0][3] Refresh failed]; nested: FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/_nyr.fdt (Too many open files)]; ]] [2013-03-06 14:30:07,003][WARN][cluster.action.shard] [Bench, Morris] received shard failed for [graylog2\_0][3], node[_-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2\_0][3] failed recovery]; nested: FlushFailedEngineException[[graylog2\_0][3] Flush failed]; nested: RefreshFailedEngineException[[graylog2\_0][3] Refresh failed]; nested: FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/\_nyr.fdt (Too many open files)]; ]] On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote: > Your index is corrupted because the max number of open files has been > exceeded. > > Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch' > > # File /etc/security/limits.conf > > elasticsearch - nofile 65535 > elasticsearch - memlock unlimited > > Jörg > > Am 05.03.13 18:27, schrieb Kranan: > > > I am using Graylog2 w/ Elasticsearch for a year now and it worked > > perfectly... since last month. > > After we changed the servers architecture (but not the server itself), > > I am experiencing repetitive > > crash of the Elasticsearch service. > > > > So I upgraded the Graylog2 to its last version (0.10.0, and of course > > Elasticsearch from the 0.19.10 to the 0.20.4 as required), but it > > still crashes every 5-6 hours, sometimes less. > > And after each crash, Elasticsearch index is gone and starting all > > over again. > > > > Here is an extract of the /var/log/elasticsearch/graylog2.log after > > the crash : > > > > ``` > > org.elasticsearch.index.engine.IndexFailedEngineException: > > [graylog2_recent][0] Index failed for > > [message#3b25a760-851f-11e2-ab8c-0024e82dd6ba] > > at > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330) > > > ``` > > at > > > > ``` > > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158) > > > ``` > > at > > > > ``` > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531) > > > ``` > > at > > > > ``` > > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429) > > > ``` > > at > > > > ``` > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > ``` > > at > > > > ``` > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > ``` > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException: > > > > ``` > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_recent/0/index/\_16bi.fdt > > > ``` > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at > > > > ``` > > java.io.RandomAccessFile.(RandomAccessFile.java:233) > > > ``` > > at > > > > ``` > > org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) > > > ``` > > at > > > > ``` > > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) > > > ``` > > at > > > > ``` > > org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) > > > ``` > > at > > org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83) > > at > > > > ``` > > org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772) > > > ``` > > at > > > > ``` > > org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577) > > > ``` > > at > > > > ``` > > org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489) > > > ``` > > ... 7 more > > > > ``` > > > > /etc/init.d/elasticsearch memory conf : > > > > ``` > > ES_MIN_MEM=256m > > ES_MAX_MEM=4g > > > > ``` > > > > Graylog2 and elasticsearch is hosted on the same server, no shard, no > > replica. > > > > ## Hope someone can help 🙂 > > > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [March 6, 2013, 6:46pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/5 "2013-03-06T18:46:26Z")
Where the changes to the conf file successfully applied? Try running 'ulimit -n' as the user running elasticsearch On Wed, Mar 6, 2013 at 6:02 AM, Kranan [nanark@gmail.com](mailto:nanark@gmail.com) wrote: > Here is the log of Elasticsearch booting up : > > [2013-03-06 14:29:46,484][INFO][node] [Bench, > Morris] {0.20.4}[7148]: initializing ... > [2013-03-06 14:29:46,511][INFO][plugins] [Bench, > Morris] loaded , sites > [2013-03-06 14:29:50,383][INFO][node] [Bench, > Morris] {0.20.4}[7148]: initialized > [2013-03-06 14:29:50,383][INFO][node] [Bench, > Morris] {0.20.4}[7148]: starting ... > [2013-03-06 14:29:50,545][INFO][transport] [Bench, > Morris] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address > {inet[/xxx.xxx.xxx.150:9300]} > [2013-03-06 14:29:53,629][INFO][cluster.service] [Bench, > Morris] new\_master [Bench, > Morris][_-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason: > zen-disco-join (elected\_as\_master) > [2013-03-06 14:29:53,636][INFO][discovery] [Bench, > Morris] graylog2/_-ugh0MlRdymGhCE8gYl-w > [2013-03-06 14:29:53,707][INFO][http] [Bench, > Morris] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address > {inet[/xxx.xxx.xxx.150:9200]} > [2013-03-06 14:29:53,707][INFO][node] [Bench, > Morris] {0.20.4}[7148]: started > [2013-03-06 14:30:00,244][INFO][gateway] [Bench, > Morris] recovered [2] indices into cluster\_state > [2013-03-06 14:30:00,267][INFO][cluster.service] [Bench, > Morris] added > {[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true, > data=false, master=false},}, reason: zen-disco-receive(join from > node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true, > data=false, master=false}]) > [2013-03-06 14:30:06,454][WARN][index.shard.service] [Bench, > Morris] [graylog2\_0][1] Failed to perform scheduled engine refresh > org.elasticsearch.index.engine.RefreshFailedEngineException: > [graylog2\_0][1] Refresh failed > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > at > org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) > at > org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.FileNotFoundException: > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/1/index/\_nwa.nrm > (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > at > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > at > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121) > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > at > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > at > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > at > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > at > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > ... 5 more > [2013-03-06 14:30:06,459][WARN][index.merge.scheduler] [Bench, > Morris] [graylog2\_0][1] failed to merge > java.io.FileNotFoundException: > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/1/index/\_nwa.nrm > (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > at > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > at > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121) > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) > at > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) > [2013-03-06 14:30:06,499][WARN][index.shard.service] [Bench, > Morris] [graylog2\_0][2] Failed to perform scheduled engine refresh > org.elasticsearch.index.engine.RefreshFailedEngineException: > [graylog2\_0][2] Refresh failed > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > at > org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) > at > org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.FileNotFoundException: > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/2/index/\_o6g.fdx > (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > at > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > at > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133) > at > org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > at > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > at > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > at > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > at > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > ... 5 more > [2013-03-06 14:30:06,792][WARN][index.shard.service] [Bench, > Morris] [graylog2\_recent][3] Failed to perform scheduled engine refresh > org.elasticsearch.index.engine.RefreshFailedEngineException: > [graylog2\_recent][3] Refresh failed > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > at > org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) > at > org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.FileNotFoundException: > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_recent/3/index/\_m9b.frq > (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) > at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) > at > org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) > at > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) > at > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) > at > org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47) > at > org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33) > at > org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51) > at > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) > at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113) > at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70) > at > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60) > at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581) > at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587) > at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > at > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > at > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > ... 5 more > [2013-03-06 14:30:06,917][WARN][indices.cluster] [Bench, > Morris] [graylog2\_0][3] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [graylog2\_0][3] failed recovery > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: org.elasticsearch.index.engine.FlushFailedEngineException: > [graylog2\_0][3] Flush failed > at > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > ... 3 more > Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException: > [graylog2\_0][3] Refresh failed > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > at > org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967) > at > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892) > ... 6 more > Caused by: java.io.FileNotFoundException: > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/_nyr.fdt > (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > at > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > at > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131) > at > org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) > at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > at > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > at > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > at > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > at > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > at > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > at > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > at > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > ... 8 more > [2013-03-06 14:30:07,003][WARN][cluster.action.shard] [Bench, > Morris] sending failed shard for [graylog2\_0][3], > node[_-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start > shard, message [IndexShardGatewayRecoveryException[[graylog2\_0][3] failed > recovery]; nested: FlushFailedEngineException[[graylog2\_0][3] Flush > failed]; nested: RefreshFailedEngineException[[graylog2\_0][3] Refresh > failed]; nested: > FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/_nyr.fdt > (Too many open files)]; ]] > [2013-03-06 14:30:07,003][WARN][cluster.action.shard] [Bench, > Morris] received shard failed for [graylog2\_0][3], > node[_-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start > shard, message [IndexShardGatewayRecoveryException[[graylog2\_0][3] failed > recovery]; nested: FlushFailedEngineException[[graylog2\_0][3] Flush > failed]; nested: RefreshFailedEngineException[[graylog2\_0][3] Refresh > failed]; nested: > FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/\_nyr.fdt > (Too many open files)]; ]] > > On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote: > > > Your index is corrupted because the max number of open files has been > > exceeded. > > > > Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch' > > > > # File /etc/security/limits.conf > > > > elasticsearch - nofile 65535 > > elasticsearch - memlock unlimited > > > > Jörg > > > > Am 05.03.13 18:27, schrieb Kranan: > > > > > I am using Graylog2 w/ Elasticsearch for a year now and it worked > > > perfectly... since last month. > > > After we changed the servers architecture (but not the server itself), > > > I am experiencing repetitive > > > crash of the Elasticsearch service. > > > > > > So I upgraded the Graylog2 to its last version (0.10.0, and of course > > > Elasticsearch from the 0.19.10 to the 0.20.4 as required), but it > > > still crashes every 5-6 hours, sometimes less. > > > And after each crash, Elasticsearch index is gone and starting all > > > over again. > > > > > > Here is an extract of the /var/log/elasticsearch/\*\*graylog2.log after > > > the crash : > > > > > > ``` > > > org.elasticsearch.index. **engine.** IndexFailedEngineException: > > > [graylog2_recent][0] Index failed for > > > [message#3b25a760-851f-11e2-**ab8c-0024e82dd6ba] > > > at > > > org.elasticsearch.index. **engine.robin.RobinEngine.** index(RobinEngine.java:494) > > > > > > ``` > > > > > ``` > > > at > > > org.elasticsearch.index.shard. **service.InternalIndexShard.** > > > > > > ``` > > > > index(InternalIndexShard.java:\*\*330) > > > > > ``` > > > at > > > org.elasticsearch.action.bulk. **TransportShardBulkAction.** > > > > > > ``` > > > > shardOperationOnPrimary(\*\*TransportShardBulkAction.java:\*\*158) > > > > > ``` > > > at > > > org.elasticsearch.action. **support.replication.** > > > > > > ``` > > > > TransportShardReplicationOpera\*\*tionAction$ **AsyncShardOperationAction.** > > performOnPrimary( **TransportShardReplicationOpera** tionAction.java:531) > > > > > ``` > > > at > > > org.elasticsearch.action. **support.replication.** > > > > > > ``` > > > > TransportShardReplicationOpera\*\*tionAction$\*\*AsyncShardOperationAction$1. > > \*\*run( **TransportShardReplicationOpera** tionAction.java:429) > > > > > ``` > > > at > > > java.util.concurrent.**ThreadPoolExecutor.runWorker(**ThreadPoolExecutor.java:1110) > > > > > > ``` > > > > > ``` > > > at > > > java.util.concurrent.**ThreadPoolExecutor$Worker.run(**ThreadPoolExecutor.java:603) > > > > > > ``` > > > > > ``` > > > at java.lang.Thread.run(Thread.**java:636) > > > Caused by: java.io.FileNotFoundException: > > > /scdata.local/syslog/ **elasticsearch/graylog2/nodes/** > > > > > > ``` > > > > 0/indices/graylog2\_recent/0/\*\*index/\_16bi.fdt > > > > > ``` > > > (Too many open files) > > > at java.io.RandomAccessFile.open(**Native Method) > > > at java.io.RandomAccessFile.<**init>(RandomAccessFile.java: > > > > > > ``` > > > > \*\*233) > > > > > ``` > > > at > > > org.apache.lucene.store. **FSDirectory$FSIndexOutput.<** init>(FSDirectory.java:441) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.store.**FSDirectory.createOutput(**FSDirectory.java:306) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.store.**XNIOFSDirectory.createOutput(**XNIOFSDirectory.java:48) > > > > > > ``` > > > > > ``` > > > at > > > org.elasticsearch.index.store. **Store$StoreDirectory.** createOutput(Store.java:487) > > > > > > ``` > > > > > ``` > > > at > > > org.elasticsearch.index.store. **Store$StoreDirectory.** createOutput(Store.java:459) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index.**FieldsWriter.(**FieldsWriter.java:83) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index. **StoredFieldsWriter.** initFieldsWriter(**StoredFieldsWriter.java:64) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index. **StoredFieldsWriter.** finishDocument(**StoredFieldsWriter.java:107) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index. **StoredFieldsWriter$PerDoc.** > > > > > > ``` > > > > finish(StoredFieldsWriter.\*\*java:151) > > > > > ``` > > > at > > > org.apache.lucene.index. **DocumentsWriter$WaitQueue.** > > > > > > ``` > > > > writeDocument(DocumentsWriter.\*\*java:1404) > > > > > ``` > > > at > > > org.apache.lucene.index.**DocumentsWriter$WaitQueue.add(**DocumentsWriter.java:1424) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index. **DocumentsWriter.** finishDocument(**DocumentsWriter.java:1043) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index. **DocumentsWriter.** updateDocument(**DocumentsWriter.java:772) > > > > > > ``` > > > > > ``` > > > at > > > org.apache.lucene.index.**IndexWriter.addDocument(**IndexWriter.java:2060) > > > > > > ``` > > > > > ``` > > > at > > > org.elasticsearch.index. **engine.robin.RobinEngine.** > > > > > > ``` > > > > innerIndex(RobinEngine.java:\*\*577) > > > > > ``` > > > at > > > org.elasticsearch.index. **engine.robin.RobinEngine.** index(RobinEngine.java:489) > > > > > > ``` > > > > > ``` > > > ... 7 more > > > > > > ``` > > > > > > /etc/init.d/elasticsearch memory conf : > > > > > > ``` > > > ES_MIN_MEM=256m > > > ES_MAX_MEM=4g > > > > > > ``` > > > > > > Graylog2 and elasticsearch is hosted on the same server, no shard, no > > > replica. > > > > > > ## Hope someone can help 🙂 > > > > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Kranan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kranan/32/2455_2.png) [@Kranan](https://discuss.elastic.co/u/Kranan)\ **Post date:** [March 6, 2013, 9:05pm UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/6 "2013-03-06T21:05:32Z")
Ok, great. So far it works fine with the nofile parameter only. Thanx. On Wednesday, March 6, 2013 7:46:26 PM UTC+1, Ivan Brusic wrote: > Where the changes to the conf file successfully applied? > > Try running 'ulimit -n' as the user running elasticsearch > > On Wed, Mar 6, 2013 at 6:02 AM, Kranan \<[nan...@gmail.com](mailto:nan...@gmail.com) \\>wrote: > > > Here is the log of Elasticsearch booting up : > > > > [2013-03-06 14:29:46,484][INFO][node] [Bench, > > Morris] {0.20.4}[7148]: initializing ... > > [2013-03-06 14:29:46,511][INFO][plugins] [Bench, > > Morris] loaded , sites > > [2013-03-06 14:29:50,383][INFO][node] [Bench, > > Morris] {0.20.4}[7148]: initialized > > [2013-03-06 14:29:50,383][INFO][node] [Bench, > > Morris] {0.20.4}[7148]: starting ... > > [2013-03-06 14:29:50,545][INFO][transport] [Bench, > > Morris] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address > > {inet[/xxx.xxx.xxx.150:9300]} > > [2013-03-06 14:29:53,629][INFO][cluster.service] [Bench, > > Morris] new\_master [Bench, > > Morris][_-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason: > > zen-disco-join (elected\_as\_master) > > [2013-03-06 14:29:53,636][INFO][discovery] [Bench, > > Morris] graylog2/_-ugh0MlRdymGhCE8gYl-w > > [2013-03-06 14:29:53,707][INFO][http] [Bench, > > Morris] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address > > {inet[/xxx.xxx.xxx.150:9200]} > > [2013-03-06 14:29:53,707][INFO][node] [Bench, > > Morris] {0.20.4}[7148]: started > > [2013-03-06 14:30:00,244][INFO][gateway] [Bench, > > Morris] recovered [2] indices into cluster\_state > > [2013-03-06 14:30:00,267][INFO][cluster.service] [Bench, > > Morris] added > > {[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true, > > data=false, master=false},}, reason: zen-disco-receive(join from > > node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true, > > data=false, master=false}]) > > [2013-03-06 14:30:06,454][WARN][index.shard.service] [Bench, > > Morris] [graylog2\_0][1] Failed to perform scheduled engine refresh > > org.elasticsearch.index.engine.RefreshFailedEngineException: > > [graylog2\_0][1] Refresh failed > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException: > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/1/index/\_nwa.nrm > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > > at > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > > at > > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > > at > > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > > at > > org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > at > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > at > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > at > > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > > at > > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > > ... 5 more > > [2013-03-06 14:30:06,459][WARN][index.merge.scheduler] [Bench, > > Morris] [graylog2\_0][1] failed to merge > > java.io.FileNotFoundException: > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/1/index/\_nwa.nrm > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > > at > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > > at > > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > > at > > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > > at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > at > > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238) > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) > > at > > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) > > at > > org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) > > at > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) > > [2013-03-06 14:30:06,499][WARN][index.shard.service] [Bench, > > Morris] [graylog2\_0][2] Failed to perform scheduled engine refresh > > org.elasticsearch.index.engine.RefreshFailedEngineException: > > [graylog2\_0][2] Refresh failed > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException: > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/2/index/\_o6g.fdx > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > > at > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > > at > > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > > at > > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > > at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133) > > at > > org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > at > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > at > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > at > > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > > at > > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > > ... 5 more > > [2013-03-06 14:30:06,792][WARN][index.shard.service] [Bench, > > Morris] [graylog2\_recent][3] Failed to perform scheduled engine refresh > > org.elasticsearch.index.engine.RefreshFailedEngineException: > > [graylog2\_recent][3] Refresh failed > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.FileNotFoundException: > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_recent/3/index/\_m9b.frq > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > > at > > org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441) > > at > > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306) > > at > > org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48) > > at > > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487) > > at > > org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459) > > at > > org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47) > > at > > org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33) > > at > > org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51) > > at > > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85) > > at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113) > > at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70) > > at > > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60) > > at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581) > > at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587) > > at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > at > > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > > at > > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > > ... 5 more > > [2013-03-06 14:30:06,917][WARN][indices.cluster] [Bench, > > Morris] [graylog2\_0][3] failed to start shard > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > [graylog2\_0][3] failed recovery > > at > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: org.elasticsearch.index.engine.FlushFailedEngineException: > > [graylog2\_0][3] Flush failed > > at > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909) > > at > > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564) > > at > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231) > > at > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > > ... 3 more > > Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException: > > [graylog2\_0][3] Refresh failed > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892) > > ... 6 more > > Caused by: java.io.FileNotFoundException: > > /scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/_nyr.fdt > > (Too many open files) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71) > > at > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98) > > at > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92) > > at > > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79) > > at > > org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537) > > at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131) > > at > > org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) > > at > > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654) > > at > > org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142) > > at > > org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451) > > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399) > > at > > org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432) > > at > > org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375) > > at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109) > > at > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57) > > at > > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137) > > at > > org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769) > > ... 8 more > > [2013-03-06 14:30:07,003][WARN][cluster.action.shard] [Bench, > > Morris] sending failed shard for [graylog2\_0][3], > > node[_-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start > > shard, message [IndexShardGatewayRecoveryException[[graylog2\_0][3] failed > > recovery]; nested: FlushFailedEngineException[[graylog2\_0][3] Flush > > failed]; nested: RefreshFailedEngineException[[graylog2\_0][3] Refresh > > failed]; nested: > > FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/_nyr.fdt > > (Too many open files)]; ]] > > [2013-03-06 14:30:07,003][WARN][cluster.action.shard] [Bench, > > Morris] received shard failed for [graylog2\_0][3], > > node[_-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start > > shard, message [IndexShardGatewayRecoveryException[[graylog2\_0][3] failed > > recovery]; nested: FlushFailedEngineException[[graylog2\_0][3] Flush > > failed]; nested: RefreshFailedEngineException[[graylog2\_0][3] Refresh > > failed]; nested: > > FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2\_0/3/index/\_nyr.fdt > > (Too many open files)]; ]] > > > > On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote: > > > > > Your index is corrupted because the max number of open files has been > > > exceeded. > > > > > > Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch' > > > > > > # File /etc/security/limits.conf > > > > > > elasticsearch - nofile 65535 > > > elasticsearch - memlock unlimited > > > > > > Jörg > > > > > > Am 05.03.13 18:27, schrieb Kranan: > > > > > > > I am using Graylog2 w/ Elasticsearch for a year now and it worked > > > > perfectly... since last month. > > > > After we changed the servers architecture (but not the server itself), > > > > I am experiencing repetitive > > > > crash of the Elasticsearch service. > > > > > > > > So I upgraded the Graylog2 to its last version (0.10.0, and of course > > > > Elasticsearch from the 0.19.10 to the 0.20.4 as required), but it > > > > still crashes every 5-6 hours, sometimes less. > > > > And after each crash, Elasticsearch index is gone and starting all > > > > over again. > > > > > > > > Here is an extract of the /var/log/elasticsearch/\*\*graylog2.log after > > > > the crash : > > > > > > > > ``` > > > > org.elasticsearch.index. **engine.** IndexFailedEngineException: > > > > [graylog2_recent][0] Index failed for > > > > [message#3b25a760-851f-11e2-**ab8c-0024e82dd6ba] > > > > at > > > > org.elasticsearch.index. **engine.robin.RobinEngine.** index(RobinEngine.java:494) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.elasticsearch.index.shard. **service.InternalIndexShard.** > > > > > > > > ``` > > > > > > index(InternalIndexShard.java:\*\*330) > > > > > > > ``` > > > > at > > > > org.elasticsearch.action.bulk. **TransportShardBulkAction.** > > > > > > > > ``` > > > > > > shardOperationOnPrimary(\*\*TransportShardBulkAction.java:\*\*158) > > > > > > > ``` > > > > at > > > > org.elasticsearch.action. **support.replication.** > > > > > > > > ``` > > > > > > TransportShardReplicationOpera\*\*tionAction$\*_AsyncShardOperationAction._ > > > \*performOnPrimary( **TransportShardReplicationOpera** tionAction.java:531) > > > > > > > ``` > > > > at > > > > org.elasticsearch.action. **support.replication.** > > > > > > > > ``` > > > > > > TransportShardReplicationOpera **tionAction$** > > > AsyncShardOperationAction$1.\*\*run( **TransportShardReplicationOpera** tionAction.java:429) > > > > > > > ``` > > > > at > > > > java.util.concurrent.**ThreadPoolExecutor.runWorker(**ThreadPoolExecutor.java:1110) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > java.util.concurrent.**ThreadPoolExecutor$Worker.run(**ThreadPoolExecutor.java:603) > > > > > > > > ``` > > > > > > > ``` > > > > at java.lang.Thread.run(Thread.**java:636) > > > > Caused by: java.io.FileNotFoundException: > > > > /scdata.local/syslog/ **elasticsearch/graylog2/nodes/** > > > > > > > > ``` > > > > > > 0/indices/graylog2\_recent/0/\*\*index/\_16bi.fdt > > > > > > > ``` > > > > (Too many open files) > > > > at java.io.RandomAccessFile.open(**Native Method) > > > > at java.io.RandomAccessFile.<** > > > > > > > > ``` > > > > > > init\>(RandomAccessFile.java:\*\*233) > > > > > > > ``` > > > > at > > > > org.apache.lucene.store. **FSDirectory$FSIndexOutput.<** init>(FSDirectory.java:441) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.store.**FSDirectory.createOutput(**FSDirectory.java:306) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.store.**XNIOFSDirectory.createOutput(**XNIOFSDirectory.java:48) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.elasticsearch.index.store. **Store$StoreDirectory.** createOutput(Store.java:487) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.elasticsearch.index.store. **Store$StoreDirectory.** createOutput(Store.java:459) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index.**FieldsWriter.(**FieldsWriter.java:83) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index. **StoredFieldsWriter.** initFieldsWriter(**StoredFieldsWriter.java:64) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index. **StoredFieldsWriter.** finishDocument(**StoredFieldsWriter.java:107) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index. **StoredFieldsWriter$PerDoc.** > > > > > > > > ``` > > > > > > finish(StoredFieldsWriter.\*\*java:151) > > > > > > > ``` > > > > at > > > > org.apache.lucene.index. **DocumentsWriter$WaitQueue.** > > > > > > > > ``` > > > > > > writeDocument(DocumentsWriter.\*\*java:1404) > > > > > > > ``` > > > > at > > > > org.apache.lucene.index.**DocumentsWriter$WaitQueue.add(**DocumentsWriter.java:1424) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index. **DocumentsWriter.** finishDocument(**DocumentsWriter.java:1043) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index. **DocumentsWriter.** updateDocument(**DocumentsWriter.java:772) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.apache.lucene.index.**IndexWriter.addDocument(**IndexWriter.java:2060) > > > > > > > > ``` > > > > > > > ``` > > > > at > > > > org.elasticsearch.index. **engine.robin.RobinEngine.** > > > > > > > > ``` > > > > > > innerIndex(RobinEngine.java:\*\*577) > > > > > > > ``` > > > > at > > > > org.elasticsearch.index. **engine.robin.RobinEngine.** index(RobinEngine.java:489) > > > > > > > > ``` > > > > > > > ``` > > > > ... 7 more > > > > > > > > ``` > > > > > > > > /etc/init.d/elasticsearch memory conf : > > > > > > > > ``` > > > > ES_MIN_MEM=256m > > > > ES_MAX_MEM=4g > > > > > > > > ``` > > > > > > > > Graylog2 and elasticsearch is hosted on the same server, no shard, no > > > > replica. > > > > > > > > ## Hope someone can help 🙂 > > > > > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/repetitive-indexfailedengineexception-crash/11019/7 "2017-07-06T02:47:58Z")