# 0.20.5 -\> 0.90.5 data migration causes: failed to merge -\> failed engine -\> docs out of order
**URL:** https://discuss.elastic.co/t/0-20-5-0-90-5-data-migration-causes-failed-to-merge-failed-engine-docs-out-of-order/14203
**Category:** Elasticsearch
**Created:** [October 31, 2013, 9:36pm UTC](https://discuss.elastic.co/t/0-20-5-0-90-5-data-migration-causes-failed-to-merge-failed-engine-docs-out-of-order/14203 "2013-10-31T21:36:32Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@george\_monroe](https://discuss.elastic.co/u/george_monroe)
#### Post date: [October 31, 2013, 9:36pm UTC](https://discuss.elastic.co/t/0-20-5-0-90-5-data-migration-causes-failed-to-merge-failed-engine-docs-out-of-order/14203/1 "2013-10-31T21:36:32Z")
Hi,
Just upgraded to 0.90.5 and migrated the data folder from 0.20.5. At
first, cluster health was yellow, but as soon as we indexed some more docs,
the node lost a shard and we are seeing these exceptions:
What could be causing these?? (I had to manually tinker with cluster names
old vs. new - could this cause this?)
Thanks!
[2013-10-31 00:57:56,382][INFO][discovery]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)]
ocho-intdev/LHxQcYFSRt2OkPj1W2iNew
[2013-10-31 00:57:56,770][INFO][http]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] bound\_address
{inet[/0.0.0.0:9200]}, publish\_address {inet[/xxx.xx.x.xxx:9200]}
[2013-10-31 00:57:56,770][INFO][node]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] started
[2013-10-31 00:57:57,060][INFO][gateway]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] recovered [1]
indices into cluster\_state
[2013-10-31 00:58:40,650][INFO][cluster.metadata]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] [ocho] opening index
[2013-10-31 13:48:19,422][WARN][index.merge.scheduler]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] [ocho][3] failed to
merge
org.apache.lucene.index.CorruptIndexException: docs out of order (5 \<= 5 )
(docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)
at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-10-31 13:48:19,617][WARN][index.engine.robin]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] [ocho][3] failed
engine
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: docs out of order (5 \<= 5 )
(docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutpu
t@f1abb1c)
at
org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:99)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order
(5 \<= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)
at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-10-31 13:48:20,037][WARN][cluster.action.shard]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] sending failed shard
for [ocho][3], node[LHxQcYFSRt2OkPj1W2iNew], [P], s[STARTED], reason
[engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (5 \<= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)
]; nested: CorruptIndexException[docs out of order (5 \<= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)];
]]
[2013-10-31 13:48:20,037][WARN][cluster.action.shard]
[[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com](http://ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com)] received shard
failed for [ocho][3], node[LHxQcYFSRt2OkPj1W2iNew], [P], s[STARTED], reaso
n [engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (5 \<= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c
)]; nested: CorruptIndexException[docs out of order (5 \<= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)];
]]
[root@#################### logs]#
rg.apache.lucene.index.CorruptIndexException: docs out of
-bash: rg.apache.lucene.index.CorruptIndexException:: command not found
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:09am UTC](https://discuss.elastic.co/t/0-20-5-0-90-5-data-migration-causes-failed-to-merge-failed-engine-docs-out-of-order/14203/2 "2017-07-06T02:09:32Z")