The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6

kabali12345 · January 11, 2018, 4:55pm

Hi all....

i have updated my ES cluster from 5.4 to 6.0.0 version recently,
Now i am trying to shrink the index with curator scripts but i am getting following error


[2018-01-10T20:26:13,358][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_2] [archiveaws_dynamodb-2017.52] creating index, cause [shrink_index], templates [], shards [1]/[2], mappings []
[2018-01-10T20:26:13,498][INFO ][o.e.c.m.MetaDataMappingService] [node_2] [archiveaws_dynamodb-2017.52/_NOPeWWcRnSTHlwXp3LM_g] create_mapping [metrics/aws/dynamodb/values]
[2018-01-10T20:26:13,588][WARN ][o.e.c.a.s.ShardStateAction] [node_2] [archiveaws_dynamodb-2017.52][0] received shard failed for shard id [[archiveaws_dynamodb-2017.52][0]], allocation id [SN7DB7OpTJC6eX283vcGyA], primary term [0], message [failed recovery], failure [RecoveryFailedException[[archiveaws_dynamodb-2017.52][0]: Recovery failed on {node_1}{CAP8ur1HSWCzer84WnBy9Q}{zi7eIPs8QkaKmLaiFs-Auw}{10.0.2.102}{10.0.2.102:9344}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [archiveaws_dynamodb-2017.52][0]: Recovery failed on {node_1}{CAP8ur1HSWCzer84WnBy9Q}{zi7eIPs8QkaKmLaiFs-Auw}{10.0.2.102}{10.0.2.102:9344}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2055) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-6.0.0.jar:6.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:328) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:117) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1530) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2050) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 4 more
Caused by: java.lang.IllegalArgumentException: Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6
	at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2830) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:158) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:125) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:286) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:117) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1530) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2050) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 4 more
[2018-01-10T20:26:13,671][INFO ][o.e.c.r.a.AllocationService] [node_2] Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[archiveaws_dynamodb-2017.52][0]] ...]).
[2018-01-10T20:26:43,370][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_2] [archiveaws_dynamodb-2017.52/_NOPeWWcRnSTHlwXp3LM_g] deleting index
[2018-01-10T20:59:39,808][INFO ][o.e.m.j.JvmGcMonitorService] [node_2] [gc][634980] overhead, spent [254ms] collecting in the last [1s]
[2018-01-10T20:59:46,848][INFO ][o.e.m.j.JvmGcMonitorService] [node_2] [gc][634987] overhead, spent [286ms] collecting in the last [1s]
[2018-01-10T21:59:39,567][INFO ][o.e.m.j.JvmGcMonitorService] [node_2] [gc][638577] overhead, spent [310ms] collecting in the last [1s]

So when i searched for it i got following link but i am not sure how make use of it, So i need your help to resolve this.
[https://github.com/elastic/elasticsearch/issues/28061]

Thank You.

dadoonet · January 11, 2018, 5:08pm

You probably need to rewrite the segments first. Calling the forcemerge API should do it.

My 2 cents.

colings86 · January 11, 2018, 6:06pm

The link you point to is a bug report that has been fixed by this Pull Request: https://github.com/elastic/elasticsearch/pull/28076

The fix will be available from the 6.1.2 release. Until then a workaround would be to call forcemerge on the index before shrinking

kabali12345 · January 11, 2018, 7:02pm

Hello thank you...
i did force forcemerge on index aws_dynamodb-2017.52 with following command

curl -XPOST "http://10.0.2.102:9233/aws_dynamodb-2017.52/_forcemerge?max_num_segments=1 "

and segments are merged to 1
then i have tried Shrinking that index with curator script where i am seeing following error

[ec2-user@ip-10-0-2-125 curator]$ curator --config curator.yml shrink.yml
2018-01-11 18:52:02,255 INFO      Preparing Action ID: 1, "shrink"
2018-01-11 18:52:02,258 INFO      Trying Action ID: 1, "shrink": Shrink logstash indices older than 21 days on the node with the most available space, excluding the node named 'not_this_node'. Delete each source index after successful shrink, then reroute the shrunk index with the provided parameters.
2018-01-11 18:52:03,309 INFO      Source index: aws_dynamodb-2017.52 -- Target index: archiveaws_dynamodb-2017.52
2018-01-11 18:52:03,888 WARNING   Not skipping node "node_2" which is a master node (not recommended), but permit_masters is True
2018-01-11 18:52:04,238 WARNING   Not skipping node "node_5" which is a master node (not recommended), but permit_masters is True
2018-01-11 18:52:04,573 WARNING   Not skipping node "node_1" which is a master node (not recommended), but permit_masters is True
2018-01-11 18:52:04,785 INFO      Moving shards to shrink node: "node_5"
2018-01-11 18:52:04,932 INFO      Health Check for all provided keys passed.
2018-01-11 18:52:05,108 INFO      Health Check for all provided keys passed.
2018-01-11 18:52:05,109 INFO      Shrinking index "aws_dynamodb-2017.52" to "archiveaws_dynamodb-2017.52" with settings: {'settings': {'index.codec': 'best_compression', 'index.number_of_replicas': 2, 'index.number_of_shards': 1}}, wait_for_active_shards=1
2018-01-11 18:52:35,141 ERROR     Deleting target index "archiveaws_dynamodb-2017.52" due to failure to complete shrink
2018-01-11 18:52:35,438 ERROR     Failed to complete action: shrink.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: global name 'index' is not defined

and in elasticsearch logs i have seen following error

[2018-01-11T18:52:05,112][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_2] [archiveaws_dynamodb-2017.52] creating index, cause [shrink_index], templates [], shards [1]/[2], mappings []
[2018-01-11T18:52:05,281][INFO ][o.e.c.m.MetaDataMappingService] [node_2] [archiveaws_dynamodb-2017.52/SmWLnuFDRYSCMWZDDvH5oQ] create_mapping [metrics/aws/dynamodb/values]
[2018-01-11T18:52:05,337][WARN ][o.e.c.a.s.ShardStateAction] [node_2] [archiveaws_dynamodb-2017.52][0] received shard failed for shard id [[archiveaws_dynamodb-2017.52][0]], allocation id [JVMOX-CaSZOSb95Gk8NkJQ], primary term [0], message [failed recovery], failure [RecoveryFailedException[[archiveaws_dynamodb-2017.52][0]: Recovery failed on {node_5}{JjDRQ5AiTa6mphTS_QhSiQ}{o7EDxtjjQySxJTPBLlCevg}{10.0.2.XXX}{10.0.2.XXX:9XXX}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [archiveaws_dynamodb-2017.52][0]: Recovery failed on {node_5}{JjDRQ5AiTa6mphTS_QhSiQ}{o7EDxtjjQySxJTPBLlCevg}{10.0.2.XXX}{10.0.2.XXX:9XXX}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2055) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-6.0.0.jar:6.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:328) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:117) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1530) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2050) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 4 more
Caused by: java.lang.IllegalArgumentException: Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6
	at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2830) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35]
	at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:158) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:125) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:286) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:117) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1530) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$8(IndexShard.java:2050) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 4 more

kabali12345 · January 16, 2018, 4:53pm

Hi all.....
am i did anything wrong here?

kabali12345 · January 18, 2018, 6:32pm

Hi all........
i would like to have your suggestions on this error?

system · February 15, 2018, 6:32pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.