All shards failed exception during reindexing kibana index

Hi,
I am running ELK 7.0.1 cluster in kubernetes environment.
I had few visualizations stored in .kibana_1 index. I have taken the snapshot of this.
Later, .kibana_1 index was deleted & it got recreated when kibana was restarted. The new .kibana has some other saved objects(dashboards, index patterns) etc.

I restored from the snapshot & renamed the restored index as .kibana_1-restored. Now, I have .kibana alias pointing to two .kibana indices.

green open .kibana_1
green open .kibana_1-restored_1565373769

Because I can't load kibana UI when .kibana alias points to two indices, I tried merging the documents from both these indices into one & deleting the other. For merging, I use Re-index API as shown:

curl -HContent-Type:application/json -XPOST "http://$ELASTICSEARCH_SERVICE:$CLIENT_PORT/_reindex?pretty" -d'{
"source": {
"index": "'$source_index'"
},
"dest": {
"index": "'$dest_index'"
}
}'>

With this, I see following error logs in elasticsearch:

{"type":"log","host":"shiv-es-elasticsearch-client-699d5f4fd4-hm58q","level":"WARN","systemid":"a6aab87097d14ff5b192d56f1d73ff1a","time": "2019-08-09T13:57:01.055Z","logger":"o.e.i.r.TransportReindexAction","timezone":"UTC","marker":"[shiv-es-elasticsearch-client-699d5f4fd4-hm58q] ","log":"giving up on search because it failed with a non-retryable exception"}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:296) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:259) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:172) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11

The indices did not get reindexed as the documents from .kibana_1-restored did not get copied to .kibana_1. In this case, to access kibana, I will have to delete either of these kibana indices but this will result in loss of saved objects (on the UI).
Can you tell me if the approach to re-index kibana indices using re-index api is correct? If yes, what could be causing this error here?

Thanks,
Shivani

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.