Hi, I have been doing some testing to check how snapshots restore process works, but I’m facing some problems with .kibana index.
I have elasticsearch 6.5.4 version and .kibana indices are ".kibana_N" (incremental) type. I created snapshots saving those .kibana-N indices. When trying to simulate an elasticsearch crash in a kubernetes environment, I do the following steps:
Create a snapshot saving .kibana* indices (.kibana does not exist. When doing "curl elasticsearch:9200/_cat/indices?v" only .kibana_N type is showed)
Delete elasticsearch statefulset
Start elasticsearch statefulset in a new persistent volume
Check indices. "curl elasticsearch:9200/_cat/indices?v" shows .kibana (not .kibana_N). Of course elasticsearch and kibana versions are the same as before.
Restore the previous snapshot
In that last step, an error is raised saying "TransportError(500, 'illegal_state_exception', 'index and alias names need to be unique, but the following duplicates were found [.kibana (alias of [.kibana_1 ...]')"
The solution I found is to reindex .kibana_1 into .kibana. Everything seems to work properly, but there are no .kibana_N index anymore, so I am afraid I am changing the configuration doing that so, maybe, it would raise some errors in the future.
I would like to know why .kibana index is created after starting elasticsearch the second time, if the first one no .kibana was showed and also, Is it proper the way I am restoring .kibana_N index? or it could be an inconvenience afterwards? I would like to be able to restore .kibana_N indices keeping their original names.
Hi @tiagocosta, thank you for your quick response. I was aware of that information, but I do not understand well how I could apply that to my problem. I am quite new using Elasticsearch and Kibana, so maybe i have misunderstood the docs.
My problem was when I spin up elasticsearch again, a new .kibana index is created instead of a .kibana_N one, even when my Kibana is already upgraded (I use the same versions for Elasticsearch and Kibana than before, 6.5+). My .kibana_N indices (saved in snapshots) have .kibana as an alias, so when I try to restore them it raise and error saying .kibana (which actually exists in elasticsearch) is an alias for .kibana_N.
It seems that spinning up kibana deployment after elasticsearch sts solves the problem, .kibana_1 appears instead of .kibana. Thanks for your guidance.
@Phandora I think your original Elasticsearch/Kibana setup has a problem. If it doesn't have the .kibana index only the .kibana_N ones it could mean that in the past any migration process doesn't have completed. I think the first step we could try it's just clone your current Elasticsearch and Kibana to another instances and there test the following:
Clone Elasticsearch and Kibana to another isntances
Stop Kibana completely
Check what is your higher .kibana_N index with curl localhost:9200/_cat/indices?v
Delete your higher .kibana_N index with curl -X DELETE "localhost:9200/.kibana_N_MAX"
Start kibana and let the migration process finish before you stop kibana again.
You should now have the previous .kibana_N_MAX recreated and the .kibana index as an alias for .kibana_N_MAX
Then your backup/restore process of the indices should work
When doing curl localhost:9200/_cat/indices?v , does it should show .kibana and .kibana_N indices? I thought .kibana migrated to .kibana_N after 6.5.0 version. It is weird though that .kibana_1 does not show any aliases but, when I try to restore it, It says duplicates were found [.kibana (alias of [.kibana_1 ...]')
After your snapshot restore, the state you need to have in your kibana indices is your previous kibana_N indices + .kibana alias pointing to your higher kibana_N one.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.