Elasticsearch:primary shards lost, after server restart due to power failure on datacenter

Elastic Stack Elasticsearch
1

Hi,

we have loss the primary shard after datacenter power failure and My ES version: 2.4.6 and we have 5 data nodes ,1 master node

[2021-02-08 14:01:37,916][INFO ][cluster.service ] [es1] detected_master {es3}{5oxdhS4UQ5Gs79sAMp8jNA}{172.19.4.46}{172.19.4.46:9300}{max_local_storage_nodes=1}, added {{es3}{5oxdhS4UQ5Gs79sAMp8jNA}{172.19.4.46}{172.19.4.46:9300}{max_local_storage_nodes=1},{es4}{V6rcOJ1OTd-kscymmx6vhA}{172.19.4.48}{172.19.4.48:9300}{max_local_storage_nodes=1},{es0}{XZnVvUxdQbeukWBptsuulQ}{172.19.3.41}{172.19.3.41:9300}{max_local_storage_nodes=1},{es2}{obkkwxrlQ4KbTzOR21Ix_Q}{172.19.4.36}{172.19.4.36:9300}{max_local_storage_nodes=1},}, reason: zen-disco-receive(from master [{es3}{5oxdhS4UQ5Gs79sAMp8jNA}{172.19.4.46}{172.19.4.46:9300}{max_local_storage_nodes=1}])
[2021-02-08 14:01:38,092][WARN ][gateway ] [es1] [xforms_2016-07-07][1] shard state info found but indexUUID didn’t match expected [DvE3mV0bS9aJ2V-bl9dkkA] actual [Litr2IELSgib-l3ZbndxrQ]
[2021-02-08 14:01:38,098][WARN ][gateway ] [es1] [xforms_2016-07-07][0] shard state info found but indexUUID didn’t match expected [DvE3mV0bS9aJ2V-bl9dkkA] actual [Litr2IELSgib-l3ZbndxrQ]
[2021-02-08 14:01:38,109][WARN ][gateway ] [es1] [hqdomains_2020-02-10][2] shard state info found but indexUUID didn’t match expected [sYdD4YntSzShdieuEJhflQ] actual [98MjNAdpQ2ywJaqmIlRL_g]
[2021-02-08 14:01:38,115][WARN ][gateway ] [es1] [hqdomains_2020-02-10][0] shard state info found but indexUUID didn’t match expected [sYdD4YntSzShdieuEJhflQ] actual [98MjNAdpQ2ywJaqmIlRL_g]
[2021-02-08 14:01:38,215][INFO ][indices.store ] [es1] updating indices.store.throttle.type from [NONE] to [all]
[2021-02-08 14:01:38,215][INFO ][indices.store ] [es1] updating indices.store.throttle.max_bytes_per_sec from [10gb] to [500mb], note, type is [all]
[2021-02-08 14:01:40,618][WARN ][indices.cluster ] [es1] [[case_search_2018-05-29][4]] marking and sending shard failed due to [failed recovery]
[case_search_2018-05-29][[case_search_2018-05-29][4]] IndexShardRecoveryException[failed to fetch index version after copying it over]; nested: IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn’t, current files: []]; nested: IndexNotFoundException[no segments* file found in store(default(mmapfs(/opt/data/elasticsearch-2.4.6/data/echis-es/nodes/0/indices/case_search_2018-05-29/4/index),niofs(/opt/data/elasticsearch-2.4.6/data/echis-es/nodes/0/indices/case_search_2018-05-29/4/index))): files: []];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:224)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: [case_search_2018-05-29][[case_search_2018-05-29][4]] IndexShardRecoveryException[shard allocated for local recovery (post api), should exist, but doesn’t, current files: []]; nested: IndexNotFoundException[no segments* file found in store(default(mmapfs(/opt/data/elasticsearch-2.4.6/data/echis-es/nodes/0/indices/case_search_2018-05-29/4/index),niofs(/opt/data/elasticsearch-2.4.6/data/echis-es/nodes/0/indices/case_search_2018-05-29/4/index))): files: []];
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:208)
… 5 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(default(mmapfs(/opt/data/elasticsearch-2.4.6/data/echis-es/nodes/0/indices/case_search_2018-05-29/4/index),niofs(/opt/data/elasticsearch-2.4.6/data/echis-es/nodes/0/indices/case_search_2018-05-29/4/index))): files: []
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:726)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490)
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:95)
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:164)
at org.elasticsearch.index.store.Store.readLastCommittedSegmentsInfo(Store.java:149)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:199)
… 5 more```

Thanks,
Sanjay
2

Welcome to our community! :smiley:

2.X has been EOL for quite a number of years now, you need to upgrade as a matter of urgency.

3

Yes,Sure and we are going to upgrade ES version to 6 soon

we lost the data/primary shard, because of old version?

any chance of losing the data in the event of datacentre failure after upgrading to new version?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.