Repetitive IndexFailedEngineException crash

I am using Graylog2 w/ ElasticSearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself), I am
experiencing repetitive
crash of the ElasticSearch service.

So I upgraded the Graylog2 to its last version (0.10.0, and of course
ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it still
crashes every 5-6 hours, sometimes less.
And after each crash, ElasticSearch index is gone and starting all over
again.

Here is an extract of the /var/log/elasticsearch/graylog2.log after the
crash :

org.elasticsearch.index.engine.IndexFailedEngineException:
[graylog2_recent][0] Index failed for
[message#3b25a760-851f-11e2-ab8c-0024e82dd6ba]
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494)
at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/0/index/_16bi.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:83)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577)
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489)
... 7 more

/etc/init.d/elasticsearch memory conf :

ES_MIN_MEM=256m
ES_MAX_MEM=4g

Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.

Hope someone can help :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Your index is corrupted because the max number of open files has been
exceeded.

Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch'

File /etc/security/limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

Jörg

Am 05.03.13 18:27, schrieb Kranan:

I am using Graylog2 w/ ElasticSearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself),
I am experiencing repetitive
crash of the ElasticSearch service.

So I upgraded the Graylog2 to its last version (0.10.0, and of course
ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it
still crashes every 5-6 hours, sometimes less.
And after each crash, ElasticSearch index is gone and starting all
over again.

Here is an extract of the /var/log/elasticsearch/graylog2.log after
the crash :

org.elasticsearch.index.engine.IndexFailedEngineException:
[graylog2_recent][0] Index failed for
[message#3b25a760-851f-11e2-ab8c-0024e82dd6ba]
        at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494)
        at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330)
        at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)
        at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531)
        at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/0/index/_16bi.fdt
(Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:441)
        at
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
        at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
        at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
        at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
        at
org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83)
        at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)
        at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)
        at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)
        at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
        at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
        at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
        at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
        at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
        at
org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577)
        at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489)
        ... 7 more

/etc/init.d/elasticsearch memory conf :

ES_MIN_MEM=256m
ES_MAX_MEM=4g

Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.

Hope someone can help :slight_smile:

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanx. I tried your conf and after a few hours, it freezed the server (ssh)
for a while and erased/corrupted the index as well.

And /var/log/elasticsearch is empty.

On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote:

Your index is corrupted because the max number of open files has been
exceeded.

Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch'

File /etc/security/limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

Jörg

Am 05.03.13 18:27, schrieb Kranan:

I am using Graylog2 w/ ElasticSearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself),
I am experiencing repetitive
crash of the ElasticSearch service.

So I upgraded the Graylog2 to its last version (0.10.0, and of course
ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it
still crashes every 5-6 hours, sometimes less.
And after each crash, ElasticSearch index is gone and starting all
over again.

Here is an extract of the /var/log/elasticsearch/graylog2.log after
the crash :

org.elasticsearch.index.engine.IndexFailedEngineException: 
[graylog2_recent][0] Index failed for 
[message#3b25a760-851f-11e2-ab8c-0024e82dd6ba] 
        at 

org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494)

        at 

org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330)

        at 

org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)

        at 

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531)

        at 

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429)

        at 

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

        at 

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

        at java.lang.Thread.run(Thread.java:636) 
Caused by: java.io.FileNotFoundException: 

/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/0/index/_16bi.fdt

(Too many open files) 
        at java.io.RandomAccessFile.open(Native Method) 
        at 

java.io.RandomAccessFile.(RandomAccessFile.java:233)

        at 

org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)

        at 

org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)

        at 

org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)

        at 

org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)

        at 

org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)

        at 
org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83) 
        at 

org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)

        at 

org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)

        at 

org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)

        at 

org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)

        at 

org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)

        at 

org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)

        at 

org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)

        at 

org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)

        at 

org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577)

        at 

org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489)

        ... 7 more 

/etc/init.d/elasticsearch memory conf :

ES_MIN_MEM=256m 
ES_MAX_MEM=4g 

Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.

Hope someone can help :slight_smile:

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Here is the log of ElasticSearch booting up :

[2013-03-06 14:29:46,484][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: initializing ...
[2013-03-06 14:29:46,511][INFO ][plugins ] [Bench, Morris]
loaded [], sites []
[2013-03-06 14:29:50,383][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: initialized
[2013-03-06 14:29:50,383][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: starting ...
[2013-03-06 14:29:50,545][INFO ][transport ] [Bench, Morris]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/xxx.xxx.xxx.150:9300]}
[2013-03-06 14:29:53,629][INFO ][cluster.service ] [Bench, Morris]
new_master [Bench,
Morris][-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason:
zen-disco-join (elected_as_master)
[2013-03-06 14:29:53,636][INFO ][discovery ] [Bench, Morris]
graylog2/
-ugh0MlRdymGhCE8gYl-w
[2013-03-06 14:29:53,707][INFO ][http ] [Bench, Morris]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/xxx.xxx.xxx.150:9200]}
[2013-03-06 14:29:53,707][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: started
[2013-03-06 14:30:00,244][INFO ][gateway ] [Bench, Morris]
recovered [2] indices into cluster_state
[2013-03-06 14:30:00,267][INFO ][cluster.service ] [Bench, Morris]
added
{[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false},}, reason: zen-disco-receive(join from
node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false}])
[2013-03-06 14:30:06,454][WARN ][index.shard.service ] [Bench, Morris]
[graylog2_0][1] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][1] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,459][WARN ][index.merge.scheduler ] [Bench, Morris]
[graylog2_0][1] failed to merge
java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
[2013-03-06 14:30:06,499][WARN ][index.shard.service ] [Bench, Morris]
[graylog2_0][2] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][2] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/2/index/_o6g.fdx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,792][WARN ][index.shard.service ] [Bench, Morris]
[graylog2_recent][3] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_recent][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/3/index/_m9b.frq
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
at
org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47)
at
org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33)
at
org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,917][WARN ][indices.cluster ] [Bench, Morris]
[graylog2_0][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[graylog2_0][3] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.index.engine.FlushFailedEngineException:
[graylog2_0][3] Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
... 3 more
Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892)
... 6 more
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 8 more
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench, Morris]
sending failed shard for [graylog2_0][3], node[
-ugh0MlRdymGhCE8gYl-w],
[P], s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2_0][3] failed recovery];
nested: FlushFailedEngineException[[graylog2_0][3] Flush failed]; nested:
RefreshFailedEngineException[[graylog2_0][3] Refresh failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)]; ]]
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench, Morris]
received shard failed for [graylog2_0][3], node[
-ugh0MlRdymGhCE8gYl-w],
[P], s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2_0][3] failed recovery];
nested: FlushFailedEngineException[[graylog2_0][3] Flush failed]; nested:
RefreshFailedEngineException[[graylog2_0][3] Refresh failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/_nyr.fdt
(Too many open files)]; ]]

On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote:

Your index is corrupted because the max number of open files has been
exceeded.

Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch'

File /etc/security/limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

Jörg

Am 05.03.13 18:27, schrieb Kranan:

I am using Graylog2 w/ ElasticSearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself),
I am experiencing repetitive
crash of the ElasticSearch service.

So I upgraded the Graylog2 to its last version (0.10.0, and of course
ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it
still crashes every 5-6 hours, sometimes less.
And after each crash, ElasticSearch index is gone and starting all
over again.

Here is an extract of the /var/log/elasticsearch/graylog2.log after
the crash :

org.elasticsearch.index.engine.IndexFailedEngineException: 
[graylog2_recent][0] Index failed for 
[message#3b25a760-851f-11e2-ab8c-0024e82dd6ba] 
        at 

org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494)

        at 

org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330)

        at 

org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)

        at 

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531)

        at 

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429)

        at 

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

        at 

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

        at java.lang.Thread.run(Thread.java:636) 
Caused by: java.io.FileNotFoundException: 

/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/0/index/_16bi.fdt

(Too many open files) 
        at java.io.RandomAccessFile.open(Native Method) 
        at 

java.io.RandomAccessFile.(RandomAccessFile.java:233)

        at 

org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)

        at 

org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)

        at 

org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)

        at 

org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)

        at 

org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)

        at 
org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83) 
        at 

org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)

        at 

org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)

        at 

org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)

        at 

org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)

        at 

org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)

        at 

org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)

        at 

org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)

        at 

org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)

        at 

org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577)

        at 

org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489)

        ... 7 more 

/etc/init.d/elasticsearch memory conf :

ES_MIN_MEM=256m 
ES_MAX_MEM=4g 

Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.

Hope someone can help :slight_smile:

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Where the changes to the conf file successfully applied?

Try running 'ulimit -n' as the user running elasticsearch

On Wed, Mar 6, 2013 at 6:02 AM, Kranan nanark@gmail.com wrote:

Here is the log of ElasticSearch booting up :

[2013-03-06 14:29:46,484][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: initializing ...
[2013-03-06 14:29:46,511][INFO ][plugins ] [Bench,
Morris] loaded [], sites []
[2013-03-06 14:29:50,383][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: initialized
[2013-03-06 14:29:50,383][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: starting ...
[2013-03-06 14:29:50,545][INFO ][transport ] [Bench,
Morris] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/xxx.xxx.xxx.150:9300]}
[2013-03-06 14:29:53,629][INFO ][cluster.service ] [Bench,
Morris] new_master [Bench,
Morris][-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason:
zen-disco-join (elected_as_master)
[2013-03-06 14:29:53,636][INFO ][discovery ] [Bench,
Morris] graylog2/
-ugh0MlRdymGhCE8gYl-w
[2013-03-06 14:29:53,707][INFO ][http ] [Bench,
Morris] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/xxx.xxx.xxx.150:9200]}
[2013-03-06 14:29:53,707][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: started
[2013-03-06 14:30:00,244][INFO ][gateway ] [Bench,
Morris] recovered [2] indices into cluster_state
[2013-03-06 14:30:00,267][INFO ][cluster.service ] [Bench,
Morris] added
{[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false},}, reason: zen-disco-receive(join from
node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false}])
[2013-03-06 14:30:06,454][WARN ][index.shard.service ] [Bench,
Morris] [graylog2_0][1] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][1] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,459][WARN ][index.merge.scheduler ] [Bench,
Morris] [graylog2_0][1] failed to merge
java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
[2013-03-06 14:30:06,499][WARN ][index.shard.service ] [Bench,
Morris] [graylog2_0][2] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][2] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/2/index/_o6g.fdx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,792][WARN ][index.shard.service ] [Bench,
Morris] [graylog2_recent][3] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_recent][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/3/index/_m9b.frq
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
at
org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47)
at
org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33)
at
org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,917][WARN ][indices.cluster ] [Bench,
Morris] [graylog2_0][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[graylog2_0][3] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.index.engine.FlushFailedEngineException:
[graylog2_0][3] Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
... 3 more
Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892)
... 6 more
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 8 more
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench,
Morris] sending failed shard for [graylog2_0][3],
node[
-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[graylog2_0][3] failed
recovery]; nested: FlushFailedEngineException[[graylog2_0][3] Flush
failed]; nested: RefreshFailedEngineException[[graylog2_0][3] Refresh
failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)]; ]]
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench,
Morris] received shard failed for [graylog2_0][3],
node[
-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[graylog2_0][3] failed
recovery]; nested: FlushFailedEngineException[[graylog2_0][3] Flush
failed]; nested: RefreshFailedEngineException[[graylog2_0][3] Refresh
failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/_nyr.fdt
(Too many open files)]; ]]

On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote:

Your index is corrupted because the max number of open files has been
exceeded.

Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch'

File /etc/security/limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

Jörg

Am 05.03.13 18:27, schrieb Kranan:

I am using Graylog2 w/ ElasticSearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself),
I am experiencing repetitive
crash of the ElasticSearch service.

So I upgraded the Graylog2 to its last version (0.10.0, and of course
ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it
still crashes every 5-6 hours, sometimes less.
And after each crash, ElasticSearch index is gone and starting all
over again.

Here is an extract of the /var/log/elasticsearch/**graylog2.log after
the crash :

org.elasticsearch.index.**engine.**IndexFailedEngineException:
[graylog2_recent][0] Index failed for
[message#3b25a760-851f-11e2-**ab8c-0024e82dd6ba]
        at
org.elasticsearch.index.**engine.robin.RobinEngine.**index(RobinEngine.java:494)
        at
org.elasticsearch.index.shard.**service.InternalIndexShard.**

index(InternalIndexShard.java:**330)

        at
org.elasticsearch.action.bulk.**TransportShardBulkAction.**

shardOperationOnPrimary(**TransportShardBulkAction.java:**158)

        at
org.elasticsearch.action.**support.replication.**

TransportShardReplicationOpera**tionAction$AsyncShardOperationAction.
performOnPrimary(TransportShardReplicationOperationAction.java:531)

        at
org.elasticsearch.action.**support.replication.**

TransportShardReplicationOpera**tionAction$**AsyncShardOperationAction$1.
**run(TransportShardReplicationOperationAction.java:429)

        at
java.util.concurrent.**ThreadPoolExecutor.runWorker(**ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.**ThreadPoolExecutor$Worker.run(**ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.**java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/**elasticsearch/graylog2/nodes/**

0/indices/graylog2_recent/0/**index/_16bi.fdt

(Too many open files)
        at java.io.RandomAccessFile.open(**Native Method)
        at java.io.RandomAccessFile.<**init>(RandomAccessFile.java:

**233)

        at
org.apache.lucene.store.**FSDirectory$FSIndexOutput.<**init>(FSDirectory.java:441)
        at
org.apache.lucene.store.**FSDirectory.createOutput(**FSDirectory.java:306)
        at
org.apache.lucene.store.**XNIOFSDirectory.createOutput(**XNIOFSDirectory.java:48)
        at
org.elasticsearch.index.store.**Store$StoreDirectory.**createOutput(Store.java:487)
        at
org.elasticsearch.index.store.**Store$StoreDirectory.**createOutput(Store.java:459)
        at
org.apache.lucene.index.**FieldsWriter.<init>(**FieldsWriter.java:83)
        at
org.apache.lucene.index.**StoredFieldsWriter.**initFieldsWriter(**StoredFieldsWriter.java:64)
        at
org.apache.lucene.index.**StoredFieldsWriter.**finishDocument(**StoredFieldsWriter.java:107)
        at
org.apache.lucene.index.**StoredFieldsWriter$PerDoc.**

finish(StoredFieldsWriter.**java:151)

        at
org.apache.lucene.index.**DocumentsWriter$WaitQueue.**

writeDocument(DocumentsWriter.**java:1404)

        at
org.apache.lucene.index.**DocumentsWriter$WaitQueue.add(**DocumentsWriter.java:1424)
        at
org.apache.lucene.index.**DocumentsWriter.**finishDocument(**DocumentsWriter.java:1043)
        at
org.apache.lucene.index.**DocumentsWriter.**updateDocument(**DocumentsWriter.java:772)
        at
org.apache.lucene.index.**IndexWriter.addDocument(**IndexWriter.java:2060)
        at
org.elasticsearch.index.**engine.robin.RobinEngine.**

innerIndex(RobinEngine.java:**577)

        at
org.elasticsearch.index.**engine.robin.RobinEngine.**index(RobinEngine.java:489)
        ... 7 more

/etc/init.d/elasticsearch memory conf :

ES_MIN_MEM=256m
ES_MAX_MEM=4g

Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.

Hope someone can help :slight_smile:

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ok, great. So far it works fine with the nofile parameter only.

Thanx.

On Wednesday, March 6, 2013 7:46:26 PM UTC+1, Ivan Brusic wrote:

Where the changes to the conf file successfully applied?

Try running 'ulimit -n' as the user running elasticsearch

On Wed, Mar 6, 2013 at 6:02 AM, Kranan <nan...@gmail.com <javascript:>>wrote:

Here is the log of ElasticSearch booting up :

[2013-03-06 14:29:46,484][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: initializing ...
[2013-03-06 14:29:46,511][INFO ][plugins ] [Bench,
Morris] loaded [], sites []
[2013-03-06 14:29:50,383][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: initialized
[2013-03-06 14:29:50,383][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: starting ...
[2013-03-06 14:29:50,545][INFO ][transport ] [Bench,
Morris] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/xxx.xxx.xxx.150:9300]}
[2013-03-06 14:29:53,629][INFO ][cluster.service ] [Bench,
Morris] new_master [Bench,
Morris][-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason:
zen-disco-join (elected_as_master)
[2013-03-06 14:29:53,636][INFO ][discovery ] [Bench,
Morris] graylog2/
-ugh0MlRdymGhCE8gYl-w
[2013-03-06 14:29:53,707][INFO ][http ] [Bench,
Morris] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/xxx.xxx.xxx.150:9200]}
[2013-03-06 14:29:53,707][INFO ][node ] [Bench,
Morris] {0.20.4}[7148]: started
[2013-03-06 14:30:00,244][INFO ][gateway ] [Bench,
Morris] recovered [2] indices into cluster_state
[2013-03-06 14:30:00,267][INFO ][cluster.service ] [Bench,
Morris] added
{[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false},}, reason: zen-disco-receive(join from
node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false}])
[2013-03-06 14:30:06,454][WARN ][index.shard.service ] [Bench,
Morris] [graylog2_0][1] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][1] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at
org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,459][WARN ][index.merge.scheduler ] [Bench,
Morris] [graylog2_0][1] failed to merge
java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
[2013-03-06 14:30:06,499][WARN ][index.shard.service ] [Bench,
Morris] [graylog2_0][2] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][2] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/2/index/_o6g.fdx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,792][WARN ][index.shard.service ] [Bench,
Morris] [graylog2_recent][3] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_recent][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/3/index/_m9b.frq
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
at
org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47)
at
org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33)
at
org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,917][WARN ][indices.cluster ] [Bench,
Morris] [graylog2_0][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[graylog2_0][3] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.index.engine.FlushFailedEngineException:
[graylog2_0][3] Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
... 3 more
Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892)
... 6 more
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 8 more
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench,
Morris] sending failed shard for [graylog2_0][3],
node[
-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[graylog2_0][3] failed
recovery]; nested: FlushFailedEngineException[[graylog2_0][3] Flush
failed]; nested: RefreshFailedEngineException[[graylog2_0][3] Refresh
failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)]; ]]
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench,
Morris] received shard failed for [graylog2_0][3],
node[
-ugh0MlRdymGhCE8gYl-w], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[graylog2_0][3] failed
recovery]; nested: FlushFailedEngineException[[graylog2_0][3] Flush
failed]; nested: RefreshFailedEngineException[[graylog2_0][3] Refresh
failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/_nyr.fdt
(Too many open files)]; ]]

On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote:

Your index is corrupted because the max number of open files has been
exceeded.

Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch'

File /etc/security/limits.conf

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

Jörg

Am 05.03.13 18:27, schrieb Kranan:

I am using Graylog2 w/ ElasticSearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself),
I am experiencing repetitive
crash of the ElasticSearch service.

So I upgraded the Graylog2 to its last version (0.10.0, and of course
ElasticSearch from the 0.19.10 to the 0.20.4 as required), but it
still crashes every 5-6 hours, sometimes less.
And after each crash, ElasticSearch index is gone and starting all
over again.

Here is an extract of the /var/log/elasticsearch/**graylog2.log after
the crash :

org.elasticsearch.index.**engine.**IndexFailedEngineException: 
[graylog2_recent][0] Index failed for 
[message#3b25a760-851f-11e2-**ab8c-0024e82dd6ba] 
        at 
org.elasticsearch.index.**engine.robin.RobinEngine.**index(RobinEngine.java:494) 
        at 
org.elasticsearch.index.shard.**service.InternalIndexShard.**

index(InternalIndexShard.java:**330)

        at 
org.elasticsearch.action.bulk.**TransportShardBulkAction.**

shardOperationOnPrimary(**TransportShardBulkAction.java:**158)

        at 
org.elasticsearch.action.**support.replication.**

TransportShardReplicationOpera**tionAction$*AsyncShardOperationAction.
*performOnPrimary(TransportShardReplicationOperationAction.java:531)

        at 
org.elasticsearch.action.**support.replication.**

TransportShardReplicationOperationAction$
AsyncShardOperationAction$1.**run(TransportShardReplicationOperationAction.java:429)

        at 
java.util.concurrent.**ThreadPoolExecutor.runWorker(**ThreadPoolExecutor.java:1110) 
        at 
java.util.concurrent.**ThreadPoolExecutor$Worker.run(**ThreadPoolExecutor.java:603) 
        at java.lang.Thread.run(Thread.**java:636) 
Caused by: java.io.FileNotFoundException: 
/scdata.local/syslog/**elasticsearch/graylog2/nodes/**

0/indices/graylog2_recent/0/**index/_16bi.fdt

(Too many open files) 
        at java.io.RandomAccessFile.open(**Native Method) 
        at java.io.RandomAccessFile.<**

init>(RandomAccessFile.java:**233)

        at 
org.apache.lucene.store.**FSDirectory$FSIndexOutput.<**init>(FSDirectory.java:441) 
        at 
org.apache.lucene.store.**FSDirectory.createOutput(**FSDirectory.java:306) 
        at 
org.apache.lucene.store.**XNIOFSDirectory.createOutput(**XNIOFSDirectory.java:48) 
        at 
org.elasticsearch.index.store.**Store$StoreDirectory.**createOutput(Store.java:487) 
        at 
org.elasticsearch.index.store.**Store$StoreDirectory.**createOutput(Store.java:459) 
        at 
org.apache.lucene.index.**FieldsWriter.<init>(**FieldsWriter.java:83) 
        at 
org.apache.lucene.index.**StoredFieldsWriter.**initFieldsWriter(**StoredFieldsWriter.java:64) 
        at 
org.apache.lucene.index.**StoredFieldsWriter.**finishDocument(**StoredFieldsWriter.java:107) 
        at 
org.apache.lucene.index.**StoredFieldsWriter$PerDoc.**

finish(StoredFieldsWriter.**java:151)

        at 
org.apache.lucene.index.**DocumentsWriter$WaitQueue.**

writeDocument(DocumentsWriter.**java:1404)

        at 
org.apache.lucene.index.**DocumentsWriter$WaitQueue.add(**DocumentsWriter.java:1424) 
        at 
org.apache.lucene.index.**DocumentsWriter.**finishDocument(**DocumentsWriter.java:1043) 
        at 
org.apache.lucene.index.**DocumentsWriter.**updateDocument(**DocumentsWriter.java:772) 
        at 
org.apache.lucene.index.**IndexWriter.addDocument(**IndexWriter.java:2060) 
        at 
org.elasticsearch.index.**engine.robin.RobinEngine.**

innerIndex(RobinEngine.java:**577)

        at 
org.elasticsearch.index.**engine.robin.RobinEngine.**index(RobinEngine.java:489) 
        ... 7 more 

/etc/init.d/elasticsearch memory conf :

ES_MIN_MEM=256m 
ES_MAX_MEM=4g 

Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.

Hope someone can help :slight_smile:

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.