Docker image versions:
Elasticsearch 6.8.2
Kibana 6.8.2
Before I ask the Curator question, I have a few questions about Elasticsearch and .kibana
/.kibana_n
indices migration when Kibana starts.
Note: Before the Kibana pod restarted, I deleted all indices starting with .kibana
.
See the following log messages from the Kibana pod:
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Finished in 972ms."}
- Why is it
Creating index .kibana_2
? - Why is it
Reindexing .kibana to .kibana_1
? - Why is it
Migrating .kibana_1 saved objects to .kibana_2
? - Why is it
Pointing alias .kibana to .kibana_2
?
I ask these questions because (like this person) I am trying to restore the .kibana_1
index (i.e. saved objects, visualizations, dashboards, etc.) from a snapshot of a prod Elasticsearch to a dev Elasticsearch. But in my case, I am using Curator.
What is happening:
- When the dev Elasticsearch starts, it inevitably runs this
migrations
thing (see above logs) and I end up with.kibana_1
and.kibana_2
indices and.kibana
is an alias to.kibana_2
in dev from the start.$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/indices | grep .kibana green open .kibana_1 5YTNzmFPRVe8cxl1kfprVg 1 1 1 0 7.2kb 3.6kb green open .kibana_2 CHk5z7uJR_GuSMhg3NZAuA 1 1 3 0 24.6kb 13.6kb $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases .kibana .kibana_2 - - -
- Curator successfully closes, restores and opens the
.kibana_1
index from a snapshot made in prod to dev (note that prod only has one.kibana_n
index and it is.kibana_1
aliased as.kibana
):2020-03-20 04:45:14,773 INFO curator.actions.snapshot do_action:1781 Restoring indices "['.kibana_1']" from snapshot: kibana-20200320010015 2020-03-20 04:45:14,907 DEBUG curator.utils wait_for_it:1765 Elapsed time: 0 seconds 2020-03-20 04:45:14,911 INFO curator.utils restore_check:1608 _recovery returned an empty response. Trying again. 2020-03-20 04:45:14,911 DEBUG curator.utils wait_for_it:1768 Response: False 2020-03-20 04:45:14,911 DEBUG curator.utils wait_for_it:1788 Action "restore" not yet complete, 0 total seconds elapsed. Waiting 10 seconds before checking again. 2020-03-20 04:45:24,921 DEBUG curator.utils wait_for_it:1765 Elapsed time: 10 seconds 2020-03-20 04:45:24,927 INFO curator.utils restore_check:1611 Provided indices: ['.kibana_1'] 2020-03-20 04:45:24,927 INFO curator.utils restore_check:1612 Found indices: ['.kibana_1'] 2020-03-20 04:45:24,927 DEBUG curator.utils wait_for_it:1768 Response: True 2020-03-20 04:45:24,927 DEBUG curator.utils wait_for_it:1773 Action "restore" finished executing (may or may not have been successful) 2020-03-20 04:45:24,927 DEBUG curator.utils wait_for_it:1791 Result: True
- After the restore, I end up with the
.kibana
alias pointing to both.kibana_2
and.kibana_1
indices. So it is adding.kibana_1
to an existing alias.kibana
which already points to.kibana_2
.$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases .kibana .kibana_2 - - - .kibana .kibana_1 - - -
- Now when I
kubectl port-forward kibana
and go tohttps://localhost:5601
, I get:
and at this point my Kibana pod is at a// 20200320010704 // https://localhost:5601/app/kibana#/management/kibana/index?_g=() { "statusCode": 400, "error": "Bad Request", "message": "Alias [.kibana] has more than one indices associated with it [[.kibana_1, .kibana_2]], can't execute a single index op: [illegal_argument_exception] Alias [.kibana] has more than one indices associated with it [[.kibana_1, .kibana_2]], can't execute a single index op" }
0/1 READY
state.
Things I have tried:
-
Remove
.kibana_1
from.kibana
alias:curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d' { "actions" : [ { "remove" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }'
This resulted in an empty Kibana (i.e. no indices, saved objects, visualizations, dashboards, etc.).
-
Remove
.kibana_2
from.kibana
alias:curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d' { "actions" : [ { "remove" : { "index" : ".kibana_2", "alias" : ".kibana" } } ] }'
This resulted in the following Kibana log message:
{"type":"log","@timestamp":"2020-03-20T05:27:30Z","tags":["spaces","error"],"pid":1,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"} {"type":"response","@timestamp":"2020-03-20T05:27:30Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":302,"responseTime":19,"contentLength":9},"message":"GET /app/kibana 302 19ms - 9.0B"}
saying it is
Unable to navigate to space \"default\"
and the following error athttps://localhost:5601
:// 20200320012405 // https://localhost:5601/#/management/kibana/index?_g=() { "statusCode": 400, "error": "Bad Request", "message": "[illegal_argument_exception] application privileges must refer to at least one resource" }
I just want to have Curator restore a prod snapshot of the .kibana_1
index to dev and have it reflected in Kibana.
Any help is appreciated.