All shards failed exception during reindexing kibana index

I am running ELK 7.0.1 cluster in kubernetes environment.
I had few visualizations stored in .kibana_1 index. I have taken the snapshot of this.
Later, .kibana_1 index was deleted & it got recreated when kibana was restarted. The new .kibana has some other saved objects(dashboards, index patterns) etc.

I restored from the snapshot & renamed the restored index as .kibana_1-restored. Now, I have .kibana alias pointing to two .kibana indices.

green open .kibana_1
green open .kibana_1-restored_1565373769

Because I can't load kibana UI when .kibana alias points to two indices, I tried merging the documents from both these indices into one & deleting the other. For merging, I use Re-index API as shown:

curl -HContent-Type:application/json -XPOST "http://$ELASTICSEARCH_SERVICE:$CLIENT_PORT/_reindex?pretty" -d'{
"source": {
"index": "'$source_index'"
"dest": {
"index": "'$dest_index'"

With this, I see following error logs in elasticsearch:

{"type":"log","host":"shiv-es-elasticsearch-client-699d5f4fd4-hm58q","level":"WARN","systemid":"a6aab87097d14ff5b192d56f1d73ff1a","time": "2019-08-09T13:57:01.055Z","logger":"o.e.i.r.TransportReindexAction","timezone":"UTC","marker":"[shiv-es-elasticsearch-client-699d5f4fd4-hm58q] ","log":"giving up on search because it failed with a non-retryable exception"} all shards failed
at ~[elasticsearch-7.0.1.jar:7.0.1]
at ~[elasticsearch-7.0.1.jar:7.0.1]
at ~[elasticsearch-7.0.1.jar:7.0.1]
at ~[elasticsearch-7.0.1.jar:7.0.1]
at$performPhaseOnShard$1( ~[elasticsearch-7.0.1.jar:7.0.1]
at$1.doRun( [elasticsearch-7.0.1.jar:7.0.1]
at [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun( [elasticsearch-7.0.1.jar:7.0.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun( [elasticsearch-7.0.1.jar:7.0.1]
at [elasticsearch-7.0.1.jar:7.0.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(

The indices did not get reindexed as the documents from .kibana_1-restored did not get copied to .kibana_1. In this case, to access kibana, I will have to delete either of these kibana indices but this will result in loss of saved objects (on the UI).
Can you tell me if the approach to re-index kibana indices using re-index api is correct? If yes, what could be causing this error here?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.