Restoring deleted elasticsearch indices

Elastic Stack Elasticsearch
1

In kibana (two node cluster ) following indices got deleted from the file system and now kibana is not accessible :

bxexbuIpRXqA0n4VyXiDQA - .security-7
3R2woj_jRVCf58Q8b9--Yg - .apm-agent-configuration
RGR9AneMRx6nnQGHF0HqKg - .apm-custom-link
sA-j0eQBQf2po6HhP6AtMg - .async-search
NQ9QQDKnSEGZ9lUHu-Ezpw - .kibana_task_manager_1
_NVAa6tnTluggE2Q2x4vdw - .kibana_1
2Uw2D0vlQjiMQhZaPUkJQg - .kibana-event-log-7.8.0-000002

Status

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .apm-custom-link                RGR9AneMRx6nnQGHF0HqKg   1   1                                               
red    open   .kibana_task_manager_1          NQ9QQDKnSEGZ9lUHu-Ezpw   1   1                                               
yellow open   .security-7                     bxexbuIpRXqA0n4VyXiDQA   1   1                                               
yellow open   .async-search                   sA-j0eQBQf2po6HhP6AtMg   1   1                                               
yellow open   .apm-agent-configuration        3R2woj_jRVCf58Q8b9--Yg   1   1                                               
yellow open   .kibana-event-log-7.8.0-000002  2Uw2D0vlQjiMQhZaPUkJQg   1   1                                               
red    open   .kibana_1                       _NVAa6tnTluggE2Q2x4vdw   1   1

Seems .kibana_1 contains all the kibana settings, dashboards, etc..

Getting following error when accessing kibana in browser

{"statusCode":503,"error":"Service Unavailable","message":"No shard available for [get [.kibana][_doc][space:default]: routing [null]]: [no_shard_available_action_exception] No shard available for [get [.kibana][_doc][space:default]: routing [null]]"}

Our elasticsearch holds around 400 GB of application log indices, which are working fine. I mean application log shipping and indexing and its access via ES API is working fine. Only issue is the kibana accessibility.
Is there any way to recover these indices from file system backups ? There are no snapshot backups currently. We are having file system backup of the above indices. Is the below steps will work ? or what is the best approach to making kibana to be accessible.

  • Delete the above indices from elasticsearch using curl
  • Stop kibana
  • Restore the corresponding indices to the indices location on the file system on both elastic search nodes
  • Start kibana
2

Filesystem backups have no value but if you're sure those indices are gone then I think you can delete them using curl and then restart Kibana, without restoring anything, and it should create the missing indices again.

3

Note that you should only delete the red health indices, the yellow ones are fine and should go green eventually.

4

Thanks for your reply. We have deleted the red indices and restarted kibana on both nodes.

curl -X DELETE -k "https://127.0.0.1:9200/.kibana_1?pretty" -u xxxx
curl -X DELETE -k "https://127.0.0.1:9200/.kibana_task_manager_1?pretty" -u xxxx

Fortunately we had a json export of the kibana dashboards, visualizations etc which we were able to import and restore.

1 Like
5

That's a relief :slight_smile:

Recommend you set up proper snapshots ASAP in case next time you lose something more irreplaceable. Also don't ever delete anything from the data path by hand - this can lead to losing the whole node.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.