Elasticsearch and Curator restore action for .kibana/.kibana_n indices

Docker image versions:

Elasticsearch 6.8.2
Kibana 6.8.2

Before I ask the Curator question, I have a few questions about Elasticsearch and .kibana/.kibana_n indices migration when Kibana starts.

Note: Before the Kibana pod restarted, I deleted all indices starting with .kibana.

See the following log messages from the Kibana pod:

{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Finished in 972ms."}
  1. Why is it Creating index .kibana_2?
  2. Why is it Reindexing .kibana to .kibana_1?
  3. Why is it Migrating .kibana_1 saved objects to .kibana_2?
  4. Why is it Pointing alias .kibana to .kibana_2?

I ask these questions because (like this person) I am trying to restore the .kibana_1 index (i.e. saved objects, visualizations, dashboards, etc.) from a snapshot of a prod Elasticsearch to a dev Elasticsearch. But in my case, I am using Curator.

What is happening:

  1. When the dev Elasticsearch starts, it inevitably runs this migrations thing (see above logs) and I end up with .kibana_1 and .kibana_2 indices and .kibana is an alias to .kibana_2 in dev from the start.
    $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/indices | grep .kibana
    green open .kibana_1             5YTNzmFPRVe8cxl1kfprVg 1 1 1 0   7.2kb   3.6kb
    green open .kibana_2             CHk5z7uJR_GuSMhg3NZAuA 1 1 3 0  24.6kb  13.6kb
    $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
    .kibana   .kibana_2   - - -
  2. Curator successfully closes, restores and opens the .kibana_1 index from a snapshot made in prod to dev (note that prod only has one .kibana_n index and it is .kibana_1 aliased as .kibana):
    2020-03-20 04:45:14,773 INFO       curator.actions.snapshot           do_action:1781 Restoring indices "['.kibana_1']" from snapshot: kibana-20200320010015
    2020-03-20 04:45:14,907 DEBUG              curator.utils            wait_for_it:1765 Elapsed time: 0 seconds
    2020-03-20 04:45:14,911 INFO               curator.utils          restore_check:1608 _recovery returned an empty response. Trying again.
    2020-03-20 04:45:14,911 DEBUG              curator.utils            wait_for_it:1768 Response: False
    2020-03-20 04:45:14,911 DEBUG              curator.utils            wait_for_it:1788 Action "restore" not yet complete, 0 total seconds elapsed. Waiting 10 seconds before checking again.
    2020-03-20 04:45:24,921 DEBUG              curator.utils            wait_for_it:1765 Elapsed time: 10 seconds
    2020-03-20 04:45:24,927 INFO               curator.utils          restore_check:1611 Provided indices: ['.kibana_1']
    2020-03-20 04:45:24,927 INFO               curator.utils          restore_check:1612 Found indices: ['.kibana_1']
    2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1768 Response: True
    2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1773 Action "restore" finished executing (may or may not have been successful)
    2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1791 Result: True
  3. After the restore, I end up with the .kibana alias pointing to both .kibana_2 and .kibana_1 indices. So it is adding .kibana_1 to an existing alias .kibana which already points to .kibana_2.
    $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
    .kibana   .kibana_2   - - -
    .kibana   .kibana_1   - - -
  4. Now when I kubectl port-forward kibana and go to https://localhost:5601, I get:
    // 20200320010704
    // https://localhost:5601/app/kibana#/management/kibana/index?_g=()
      "statusCode": 400,
      "error": "Bad Request",
      "message": "Alias [.kibana] has more than one indices associated with it
    [[.kibana_1, .kibana_2]], can't execute a single index op:
    [illegal_argument_exception] Alias [.kibana] has more than one indices associated with it
    [[.kibana_1, .kibana_2]], can't execute a single index op"
    and at this point my Kibana pod is at a 0/1 READY state.

Things I have tried:

  1. Remove .kibana_1 from .kibana alias:

    curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d'
    { "actions" : [ { "remove" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }'

    This resulted in an empty Kibana (i.e. no indices, saved objects, visualizations, dashboards, etc.).

  2. Remove .kibana_2 from .kibana alias:

    curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d'
    { "actions" : [ { "remove" : { "index" : ".kibana_2", "alias" : ".kibana" } } ] }'

    This resulted in the following Kibana log message:

    {"type":"log","@timestamp":"2020-03-20T05:27:30Z","tags":["spaces","error"],"pid":1,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"}
    {"type":"response","@timestamp":"2020-03-20T05:27:30Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"","userAgent":""},"res":{"statusCode":302,"responseTime":19,"contentLength":9},"message":"GET /app/kibana 302 19ms - 9.0B"}

    saying it is Unable to navigate to space \"default\" and the following error at https://localhost:5601:

    // 20200320012405
    // https://localhost:5601/#/management/kibana/index?_g=()
      "statusCode": 400,
      "error": "Bad Request",
      "message": "[illegal_argument_exception] application privileges must refer to at
    least one resource"

I just want to have Curator restore a prod snapshot of the .kibana_1 index to dev and have it reflected in Kibana.

Any help is appreciated.

in my documention, when I want to restore prod to dev, I have this steps to do :

delete index .kibana in index management
create the alias .kibana :

POST /_aliases
    "actions" : [
        { "add" : { "index" : ".kibana-6", "alias" : ".kibana" } }

then I have my dashboards, saved search...

Hey @Ludovic9, thanks for replying. I am aware that this works from Dev Tools but I am not sure if it's the same case when using Curator.

When I try what you mentioned manually using cURL:

# since I do not have the .kibana metaindex and .kibana is an 
# alias to the .kibana_2 metaindex, I will remove the alias first
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [ { "remove": { "index": ".kibana_2", "alias": ".kibana" } } ] }'

# START NOTE once I remove the .kibana_2 metaindex from the 
# .kibana alias, ES will quickly try to recreate the default 
# .kibana index. So I will need to execute the next 2 cUrl 
# commands before ES recreates that default .kibana index END NOTE

# then I will delete the .kibana_2 metaindex
$ curl -XDELETE -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2
# then I will create the alias .kibana where .kibana_1 is the
# already restored metaindex (a snapshot of production)
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [ { "add": { "index": ".kibana_1", "alias": ".kibana" } } ] }'

Now, when I kubectl port-forward and go to https://localhost:5601, I keep getting the [illegal_argument_exception] application privileges must refer to at least one resource error due to missing the default space document. What else do I need to do to create this default space document when using Curator? I am currently just doing actions close, restore, and open in order.

Looks like I am facing this very issue that has been open since December of 2018.

So by missing this default space document, I am only able to access Discover, Visualize, Dashboard, Dev Tools, and Management.

The suggested solution is to stop the Kibana server and restart it and I don't like that idea since a restart will create downtime for the users.

I have tried manually creating this space:default document but got an error:

$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_1/_doc/space:default -H 'Content-Type: application/json' -d'
{ "space": { "name": "Default",
             "description": "This is your default space!",
             "color": "#00bfb3",
             "disabledFeatures": [],
             "_reserved": true },
  "type": "space",
  "references": [],
  "migrationVersion": { "space": "6.6.0" } }'
{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"},"status":400}

Is there a workaround?

EDIT: I realized I am trying to restore a snapshot (from a cluster without XPACK enabled) to a cluster with XPACK enabled and this restored snapshot of .kibana_n index does not have this space:default document (since this is an XPACK feature). What can be done in this case?

EDIT 2: I just tried to restore a snapshot (that I just took from the same cluster with XPACK enabled) and got the same error "[illegal_argument_exception] application privileges must refer to at least one resource". The snapshot is missing the space:default document.

The original .kibana_2 index has it defined:

$ curl -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2/_search?pretty
  "_index" : ".kibana_2",
  "_type" : "doc",
  "_id" : "space:default",
  "_score" : 1.0,
  "_source" : {
    "space" : {
      "name" : "Default",
      "description" : "This is your default space!",
      "color" : "#00bfb3",
      "_reserved" : true
    "type" : "space",
    "updated_at" : "2020-03-23T21:39:56.291Z"

but the restored .kibana_1 index does not have this doc. Why? This is preventing Curator from restoring.