Hello,
I started having this problem on September 11th, and it seemed to continue
until this morning 9/13 @7:50 local
2012-09-11 22:13:23.323 -> 2012-09-13 07:50:34.412
[2012-09-13 09:44:30,685][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2253381
<= 2253381 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@484507dc)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
After which, it seemed to change the problem document.
2012-09-13 07:52:48,636 -> Ongoing
[2012-09-13 07:52:48,636][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2260457
<= 2260457 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@6bc3da70)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
These warnings repeat every couple of minutes and seem to be causing
excessive I/O load on my disc array, which thankfully can handle the load.
I am using Elasticsearch 0.19.8 along with Graylog2 Server and Web
Interface.
I am using a single node with 20 shards and 1 replica, hosted on the same
virtual machine as Graylog2, Mongodb, and Graylog2-web-interface
I have restarted the node in an attempt to have the shards re-sync, but
this did not correct the issue.
I also suspect, unless I'm reading it wrong, that the error exists in shard
12.
All the data seems fine and I do not see any corruption or missing data.
{
- cluster_name: graylog2
- status: yellow
- timed_out: false
- number_of_nodes: 1
- number_of_data_nodes: 1
- active_primary_shards: 20
- active_shards: 20
- relocating_shards: 0
- initializing_shards: 0
- unassigned_shards: 20
}
Please let me know if you require any additional information.
Thanks in advance,
-Leigh
--
I'm going to knock on wood here, but the error stopped at 13:19:20.750
local (about 1.5 hours ago)
I will update if it starts up again.
On Thursday, 13 September 2012 10:57:03 UTC-3, Leigh Cameron wrote:
Hello,
I started having this problem on September 11th, and it seemed to continue
until this morning 9/13 @7:50 local
2012-09-11 22:13:23.323 -> 2012-09-13 07:50:34.412
[2012-09-13 09:44:30,685][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2253381
<= 2253381 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@484507dc)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
After which, it seemed to change the problem document.
2012-09-13 07:52:48,636 -> Ongoing
[2012-09-13 07:52:48,636][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2260457
<= 2260457 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@6bc3da70)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
These warnings repeat every couple of minutes and seem to be causing
excessive I/O load on my disc array, which thankfully can handle the load.
I am using Elasticsearch 0.19.8 along with Graylog2 Server and Web
Interface.
I am using a single node with 20 shards and 1 replica, hosted on the same
virtual machine as Graylog2, Mongodb, and Graylog2-web-interface
I have restarted the node in an attempt to have the shards re-sync, but
this did not correct the issue.
I also suspect, unless I'm reading it wrong, that the error exists in
shard 12.
All the data seems fine and I do not see any corruption or missing data.
{
- cluster_name: graylog2
- status: yellow
- timed_out: false
- number_of_nodes: 1
- number_of_data_nodes: 1
- active_primary_shards: 20
- active_shards: 20
- relocating_shards: 0
- initializing_shards: 0
- unassigned_shards: 20
}
Please let me know if you require any additional information.
Thanks in advance,
-Leigh
--