Discuss the Elastic Stack

Can someone confirm migration steps from 0.20.5 to 0.90.5

Elastic Stack Elasticsearch

george_monroe (george_monroe) November 1, 2013, 6:20pm 1

Hi Guys,

We are migrating data from 0.20.5 to 0.90.5 using the following steps but
we are getting merge exceptions after we try to index new data. Can
someone confirm if the basic steps are sound? What are we doing wrong?

Migration steps:

(1) stop indexing incoming data (close index)

(2) flush index

(3) shut down node (service stop)

(4) back up data folder and zip it up

(5) upgrade to ES 0.90.5

(5) transfer zip to new AWS server

(6) unzip data and copy folder to the new elasticsearch data location

(7) change cluster name (cluster name change needed because of environments)

org.apache.lucene.index.CorruptIndexException: docs out of order (287 <=
287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)

at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)

at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)

at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)

at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)

at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)

at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)

at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)

at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)

at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

[2013-10-31 17:00:16,808][WARN ][index.engine.robin ] [xxx.com]
[ocho][3] failed engine

org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: docs out of order (287 <=
287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)

at
org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:99)

at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)

Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order
(287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)

at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)

at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)

at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)

at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)

at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)

at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)

at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)

at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)

at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

[2013-10-31 17:00:16,813][WARN ][cluster.action.shard ] [xxx.com]
sending failed shard for [ocho][3], node[BHN9A7FKT6SJqJfIxeRnPA], [P],
s[STARTED], reason [engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
nested: CorruptIndexException[docs out of order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
]]

[2013-10-31 17:00:16,816][WARN ][cluster.action.shard ] [xxx.com]
received shard failed for [ocho][3], node[BHN9A7FKT6SJqJfIxeRnPA], [P],
s[STARTED], reason [engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
nested: CorruptIndexException[docs out of order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
]]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan (Ivan Brusic) November 4, 2013, 3:58pm 2

If you are upgrading from the same major version (0.90), you can do a
rolling restart if only one index at a time. I simply create a new install
directory on the same server, maintain the same data path and restart.
Startup scripts point to symlinks and not hardcoded installation
directories. With major version upgrades, I still maintain the same data
path since Lucene can read indices one version older.

You might want to ran Lucene's checkindex on your shards. Were there any
errors before the shutdown?

Cheers,

Ivan

On Fri, Nov 1, 2013 at 11:20 AM, Yuri Panchenko yuri.panchenko@gmail.comwrote:

Hi Guys,

We are migrating data from 0.20.5 to 0.90.5 using the following steps but
we are getting merge exceptions after we try to index new data. Can
someone confirm if the basic steps are sound? What are we doing wrong?

Migration steps:

(1) stop indexing incoming data (close index)

(2) flush index

(3) shut down node (service stop)

(4) back up data folder and zip it up

(5) upgrade to ES 0.90.5

(5) transfer zip to new AWS server

(6) unzip data and copy folder to the new elasticsearch data location

(7) change cluster name (cluster name change needed because of
environments)

org.apache.lucene.index.CorruptIndexException: docs out of order (287 <=
287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884
)

at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)

at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)

at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)

at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)

at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)

at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)

at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)

at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)

at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

[2013-10-31 17:00:16,808][WARN ][index.engine.robin ] [xxx.com]
[ocho][3] failed engine

org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: docs out of order (287 <=
287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884
)

at
org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:99)

at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)

Caused by: org.apache.lucene.index.CorruptIndexException: docs out of
order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884
)

at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)

at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)

at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)

at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)

at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)

at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)

at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)

at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)

at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)

at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)

at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

[2013-10-31 17:00:16,813][WARN ][cluster.action.shard ] [xxx.com]
sending failed shard for [ocho][3], node[BHN9A7FKT6SJqJfIxeRnPA], [P],
s[STARTED], reason [engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
nested: CorruptIndexException[docs out of order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
]]

[2013-10-31 17:00:16,816][WARN ][cluster.action.shard ] [xxx.com]
received shard failed for [ocho][3], node[BHN9A7FKT6SJqJfIxeRnPA], [P],
s[STARTED], reason [engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
nested: CorruptIndexException[docs out of order (287 <= 287 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@2342d884)];
]]

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views	Activity
Corrupted merge while migrating from 0.20.5 to 0.90.5 Elasticsearch	1	320	July 6, 2017
Corrupted merge when migrating from 0.20.5 to 0.90.5 Elasticsearch	1	323	July 6, 2017
0.20.5 -> 0.90.5 data migration causes: failed to merge -> failed engine -> docs out of order Elasticsearch	1	371	July 6, 2017
Index "failed to merge" Elasticsearch	1	291	July 6, 2017
Index "failed to merge" Elasticsearch	5	1898	July 6, 2017

© 2020. All Rights Reserved - Elasticsearch

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.