# Index Failed to Merge

**URL:** https://discuss.elastic.co/t/index-failed-to-merge/9012
**Category:** Elasticsearch
**Created:** [September 13, 2012, 1:57pm UTC](https://discuss.elastic.co/t/index-failed-to-merge/9012 "2012-09-13T13:57:03Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Leigh\_Cameron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leigh_cameron/32/2726_2.png) [@Leigh\_Cameron](https://discuss.elastic.co/u/Leigh_Cameron)
#### Post date: [September 13, 2012, 1:57pm UTC](https://discuss.elastic.co/t/index-failed-to-merge/9012/1 "2012-09-13T13:57:03Z")

</div>

Hello,

I started having this problem on September 11th, and it seemed to continue  
until this morning 9/13 @7:50 local  
_2012-09-11 22:13:23.323 -\> 2012-09-13 07:50:34.412_  
[2012-09-13 09:44:30,685][WARN][index.merge.scheduler] [Potts,  
Virginia "Pepper"] [graylog2][12] failed to merge  
org.apache.lucene.index.CorruptIndexException: docs out of order (2253381  
\<= 2253381 ) (out:  
org.apache.lucene.store.FSDirectory$FSIndexOutput@484507dc)  
at  
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)  
at  
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)  
at  
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)  
at  
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)  
at  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)  
at  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)  
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)  
at  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)  
at  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)  
at  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

After which, it seemed to change the problem document.  
_2012-09-13 07:52:48,636 -\> Ongoing_  
[2012-09-13 07:52:48,636][WARN][index.merge.scheduler] [Potts,  
Virginia "Pepper"] [graylog2][12] failed to merge  
org.apache.lucene.index.CorruptIndexException: docs out of order (2260457  
\<= 2260457 ) (out:  
org.apache.lucene.store.FSDirectory$FSIndexOutput@6bc3da70)  
at  
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)  
at  
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)  
at  
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)  
at  
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)  
at  
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)  
at  
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)  
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)  
at  
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)  
at  
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)  
at  
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

These warnings repeat every couple of minutes and seem to be causing  
excessive I/O load on my disc array, which thankfully can handle the load.

I am using Elasticsearch 0.19.8 along with Graylog2 Server and Web  
Interface.  
I am using a single node with 20 shards and 1 replica, hosted on the same  
virtual machine as Graylog2, Mongodb, and Graylog2-web-interface

I have restarted the node in an attempt to have the shards re-sync, but  
this did not correct the issue.  
I also suspect, unless I'm reading it wrong, that the error exists in shard  
12.  
All the data seems fine and I do not see any corruption or missing data.

{

- cluster\_name: graylog2
- status: yellow
- timed\_out: false
- number\_of\_nodes: 1
- number\_of\_data\_nodes: 1
- active\_primary\_shards: 20
- active\_shards: 20
- relocating\_shards: 0
- initializing\_shards: 0
- unassigned\_shards: 20

}

Please let me know if you require any additional information.

Thanks in advance,

-Leigh

--

---

<div class="post-metadata">

### Author: ![Leigh\_Cameron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leigh_cameron/32/2726_2.png) [@Leigh\_Cameron](https://discuss.elastic.co/u/Leigh_Cameron)
#### Post date: [September 13, 2012, 5:42pm UTC](https://discuss.elastic.co/t/index-failed-to-merge/9012/2 "2012-09-13T17:42:30Z")

</div>

I'm going to knock on wood here, but the error stopped at 13:19:20.750  
local (about 1.5 hours ago)  
I will update if it starts up again.

On Thursday, 13 September 2012 10:57:03 UTC-3, Leigh Cameron wrote:

> Hello,
> 
> I started having this problem on September 11th, and it seemed to continue  
> until this morning 9/13 @7:50 local  
> _2012-09-11 22:13:23.323 -\> 2012-09-13 07:50:34.412_  
> [2012-09-13 09:44:30,685][WARN][index.merge.scheduler] [Potts,  
> Virginia "Pepper"] [graylog2][12] failed to merge  
> org.apache.lucene.index.CorruptIndexException: docs out of order (2253381  
> \<= 2253381 ) (out:  
> org.apache.lucene.store.FSDirectory$FSIndexOutput@484507dc)  
> at  
> org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)  
> at  
> org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)  
> at  
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)  
> at  
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)  
> at  
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)  
> at  
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)  
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)  
> at  
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)  
> at  
> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)  
> at  
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
> 
> After which, it seemed to change the problem document.  
> _2012-09-13 07:52:48,636 -\> Ongoing_  
> [2012-09-13 07:52:48,636][WARN][index.merge.scheduler] [Potts,  
> Virginia "Pepper"] [graylog2][12] failed to merge  
> org.apache.lucene.index.CorruptIndexException: docs out of order (2260457  
> \<= 2260457 ) (out:  
> org.apache.lucene.store.FSDirectory$FSIndexOutput@6bc3da70)  
> at  
> org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)  
> at  
> org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)  
> at  
> org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)  
> at  
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)  
> at  
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)  
> at  
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4256)  
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3901)  
> at  
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)  
> at  
> org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)  
> at  
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
> 
> These warnings repeat every couple of minutes and seem to be causing  
> excessive I/O load on my disc array, which thankfully can handle the load.
> 
> I am using Elasticsearch 0.19.8 along with Graylog2 Server and Web  
> Interface.  
> I am using a single node with 20 shards and 1 replica, hosted on the same  
> virtual machine as Graylog2, Mongodb, and Graylog2-web-interface
> 
> I have restarted the node in an attempt to have the shards re-sync, but  
> this did not correct the issue.  
> I also suspect, unless I'm reading it wrong, that the error exists in  
> shard 12.  
> All the data seems fine and I do not see any corruption or missing data.
> 
> {
> 
> - cluster\_name: graylog2
> - status: yellow
> - timed\_out: false
> - number\_of\_nodes: 1
> - number\_of\_data\_nodes: 1
> - active\_primary\_shards: 20
> - active\_shards: 20
> - relocating\_shards: 0
> - initializing\_shards: 0
> - unassigned\_shards: 20
> 
> }
> 
> Please let me know if you require any additional information.
> 
> Thanks in advance,
> 
> -Leigh

--

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:12am UTC](https://discuss.elastic.co/t/index-failed-to-merge/9012/3 "2017-07-06T03:12:52Z")

</div>