Hi,
Just upgraded to 0.90.5 and migrated the data folder from 0.20.5. At
first, cluster health was yellow, but as soon as we indexed some more docs,
the node lost a shard and we are seeing these exceptions:
What could be causing these?? (I had to manually tinker with cluster names
old vs. new - could this cause this?)
Thanks!
[2013-10-31 00:57:56,382][INFO ][discovery ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com]
ocho-intdev/LHxQcYFSRt2OkPj1W2iNew
[2013-10-31 00:57:56,770][INFO ][http ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] bound_address
{inet[/0.0.0.0:9200]}, publish_address {inet[/xxx.xx.x.xxx:9200]}
[2013-10-31 00:57:56,770][INFO ][node ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] started
[2013-10-31 00:57:57,060][INFO ][gateway ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] recovered [1]
indices into cluster_state
[2013-10-31 00:58:40,650][INFO ][cluster.metadata ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] [ocho] opening index
[2013-10-31 13:48:19,422][WARN ][index.merge.scheduler ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] [ocho][3] failed to
merge
org.apache.lucene.index.CorruptIndexException: docs out of order (5 <= 5 )
(docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)
at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-10-31 13:48:19,617][WARN ][index.engine.robin ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] [ocho][3] failed
engine
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: docs out of order (5 <= 5 )
(docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutpu
t@f1abb1c)
at
org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:99)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order
(5 <= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.startDoc(Lucene41PostingsWriter.java:243)
at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:115)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2013-10-31 13:48:20,037][WARN ][cluster.action.shard ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] sending failed shard
for [ocho][3], node[LHxQcYFSRt2OkPj1W2iNew], [P], s[STARTED], reason
[engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (5 <= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)
]; nested: CorruptIndexException[docs out of order (5 <= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)];
]]
[2013-10-31 13:48:20,037][WARN ][cluster.action.shard ]
[ocho-intdev-newelasticsearch01.us-east-1a.dfengg.com] received shard
failed for [ocho][3], node[LHxQcYFSRt2OkPj1W2iNew], [P], s[STARTED], reaso
n [engine failure, message
[MergeException[org.apache.lucene.index.CorruptIndexException: docs out of
order (5 <= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c
)]; nested: CorruptIndexException[docs out of order (5 <= 5 ) (docOut:
org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput@f1abb1c)];
]]
[root@#################### logs]#
rg.apache.lucene.index.CorruptIndexException: docs out of
-bash: rg.apache.lucene.index.CorruptIndexException:: command not found
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.