Restoring .kibana_N indices into elasticsearch with an existing .kibana index

[Phandora (Mayte)]

Hi, I have been doing some testing to check how snapshots restore process works, but I’m facing some problems with .kibana index.

I have elasticsearch 6.5.4 version and .kibana indices are ".kibana_N" (incremental) type. I created snapshots saving those .kibana-N indices. When trying to simulate an elasticsearch crash in a kubernetes environment, I do the following steps:

• Create a snapshot saving .kibana* indices (.kibana does not exist. When doing "curl elasticsearch:9200/_cat/indices?v" only .kibana_N type is showed)
• Delete elasticsearch statefulset
• Start elasticsearch statefulset in a new persistent volume
• Check indices. "curl elasticsearch:9200/_cat/indices?v" shows .kibana (not .kibana_N). Of course elasticsearch and kibana versions are the same as before.
• Restore the previous snapshot

In that last step, an error is raised saying "TransportError(500, 'illegal_state_exception', 'index and alias names need to be unique, but the following duplicates were found [.kibana (alias of [.kibana_1 ...]')"

The solution I found is to reindex .kibana_1 into .kibana. Everything seems to work properly, but there are no .kibana_N index anymore, so I am afraid I am changing the configuration doing that so, maybe, it would raise some errors in the future.

I would like to know why .kibana index is created after starting elasticsearch the second time, if the first one no .kibana was showed and also, Is it proper the way I am restoring .kibana_N index? or it could be an inconvenience afterwards? I would like to be able to restore .kibana_N indices keeping their original names.

Any piece of advice or help appreciated.

[tiagocosta]

Hi @Phandora,

I think every information you need is summarised here on the docs:

https://www.elastic.co/guide/en/kibana/6.5/upgrade-migrations.html

Cheers

[Phandora (Mayte)]

Hi @tiagocosta, thank you for your quick response. I was aware of that information, but I do not understand well how I could apply that to my problem. I am quite new using Elasticsearch and Kibana, so maybe i have misunderstood the docs.

My problem was when I spin up elasticsearch again, a new .kibana index is created instead of a .kibana_N one, even when my Kibana is already upgraded (I use the same versions for Elasticsearch and Kibana than before, 6.5+). My .kibana_N indices (saved in snapshots) have .kibana as an alias, so when I try to restore them it raise and error saying .kibana (which actually exists in elasticsearch) is an alias for .kibana_N.

It seems that spinning up kibana deployment after elasticsearch sts solves the problem, .kibana_1 appears instead of .kibana. Thanks for your guidance.

[tiagocosta]

@Phandora I think your original Elasticsearch/Kibana setup has a problem. If it doesn't have the .kibana index only the .kibana_N ones it could mean that in the past any migration process doesn't have completed. I think the first step we could try it's just clone your current Elasticsearch and Kibana to another instances and there test the following:

• Clone Elasticsearch and Kibana to another isntances
• Stop Kibana completely
• Check what is your higher .kibana_N index with curl localhost:9200/_cat/indices?v
• Delete your higher .kibana_N index with curl -X DELETE "localhost:9200/.kibana_N_MAX"
• Start kibana and let the migration process finish before you stop kibana again.

You should now have the previous .kibana_N_MAX recreated and the .kibana index as an alias for .kibana_N_MAX

Then your backup/restore process of the indices should work

[Phandora (Mayte)]

@tiagocosta I did not attempt any migration process, this environment is fully spinned up from:

• docker.elastic.co/kibana/kibana:6.5.4

• docker.elastic.co/elasticsearch/elasticsearch:6.5.4

When doing curl localhost:9200/_cat/indices?v , does it should show .kibana and .kibana_N indices? I thought .kibana migrated to .kibana_N after 6.5.0 version. It is weird though that .kibana_1 does not show any aliases but, when I try to restore it, It says duplicates were found [.kibana (alias of [.kibana_1 ...]')

[tiagocosta]

@Phandora I thought you had already applied any migration before.

Anyway even in a blank installation for 6.5.4 you should start with .kibana_1 index and .kibana alias for .kibana_1.

One thing that I wrote wrong before was the query to get the indices and the aliases.

curl localhost:9200/_cat/aliases should return the aliases

and curl localhost:9200/_cat/indices

So in a blank installation for 6.5.4 what is expected is:

curl localhost:9200/_cat/indices
-> green open .kibana_1 ....

and

curl localhost:9200/_cat/aliases
-> .kibana .kibana_1 - - -

After your snapshot restore, the state you need to have in your kibana indices is your previous kibana_N indices + .kibana alias pointing to your higher kibana_N one.