Index Failed to Merge


(Leigh Cameron) #1

Hello,

I started having this problem on September 11th, and it seemed to continue
until this morning 9/13 @7:50 local
2012-09-11 22:13:23.323 -> 2012-09-13 07:50:34.412
[2012-09-13 09:44:30,685][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2253381
<= 2253381 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@484507dc)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

After which, it seemed to change the problem document.
2012-09-13 07:52:48,636 -> Ongoing
[2012-09-13 07:52:48,636][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2260457
<= 2260457 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@6bc3da70)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

These warnings repeat every couple of minutes and seem to be causing
excessive I/O load on my disc array, which thankfully can handle the load.

I am using Elasticsearch 0.19.8 along with Graylog2 Server and Web
Interface.
I am using a single node with 20 shards and 1 replica, hosted on the same
virtual machine as Graylog2, Mongodb, and Graylog2-web-interface

I have restarted the node in an attempt to have the shards re-sync, but
this did not correct the issue.
I also suspect, unless I'm reading it wrong, that the error exists in shard
12.
All the data seems fine and I do not see any corruption or missing data.

{

  • cluster_name: graylog2
  • status: yellow
  • timed_out: false
  • number_of_nodes: 1
  • number_of_data_nodes: 1
  • active_primary_shards: 20
  • active_shards: 20
  • relocating_shards: 0
  • initializing_shards: 0
  • unassigned_shards: 20

}

Please let me know if you require any additional information.

Thanks in advance,

-Leigh

--


(Leigh Cameron) #2

I'm going to knock on wood here, but the error stopped at 13:19:20.750
local (about 1.5 hours ago)
I will update if it starts up again.

On Thursday, 13 September 2012 10:57:03 UTC-3, Leigh Cameron wrote:

Hello,

I started having this problem on September 11th, and it seemed to continue
until this morning 9/13 @7:50 local
2012-09-11 22:13:23.323 -> 2012-09-13 07:50:34.412
[2012-09-13 09:44:30,685][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2253381
<= 2253381 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@484507dc)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

After which, it seemed to change the problem document.
2012-09-13 07:52:48,636 -> Ongoing
[2012-09-13 07:52:48,636][WARN ][index.merge.scheduler ] [Potts,
Virginia "Pepper"] [graylog2][12] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (2260457
<= 2260457 ) (out:
org.apache.lucene.store.FSDirectory$FSIndexOutput@6bc3da70)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

These warnings repeat every couple of minutes and seem to be causing
excessive I/O load on my disc array, which thankfully can handle the load.

I am using Elasticsearch 0.19.8 along with Graylog2 Server and Web
Interface.
I am using a single node with 20 shards and 1 replica, hosted on the same
virtual machine as Graylog2, Mongodb, and Graylog2-web-interface

I have restarted the node in an attempt to have the shards re-sync, but
this did not correct the issue.
I also suspect, unless I'm reading it wrong, that the error exists in
shard 12.
All the data seems fine and I do not see any corruption or missing data.

{

  • cluster_name: graylog2
  • status: yellow
  • timed_out: false
  • number_of_nodes: 1
  • number_of_data_nodes: 1
  • active_primary_shards: 20
  • active_shards: 20
  • relocating_shards: 0
  • initializing_shards: 0
  • unassigned_shards: 20

}

Please let me know if you require any additional information.

Thanks in advance,

-Leigh

--


(system) #3