Elasticsearch and Curator restore action for .kibana/.kibana_n indices

Docker image versions:

Elasticsearch 6.8.2
Kibana 6.8.2

Before I ask the Curator question, I have a few questions about Elasticsearch and .kibana/.kibana_n indices migration when Kibana starts.

Note: Before the Kibana pod restarted, I deleted all indices starting with .kibana.

See the following log messages from the Kibana pod:

{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Finished in 972ms."}
  1. Why is it Creating index .kibana_2?
  2. Why is it Reindexing .kibana to .kibana_1?
  3. Why is it Migrating .kibana_1 saved objects to .kibana_2?
  4. Why is it Pointing alias .kibana to .kibana_2?

I ask these questions because (like this person) I am trying to restore the .kibana_1 index (i.e. saved objects, visualizations, dashboards, etc.) from a snapshot of a prod Elasticsearch to a dev Elasticsearch. But in my case, I am using Curator.

What is happening:

  1. When the dev Elasticsearch starts, it inevitably runs this migrations thing (see above logs) and I end up with .kibana_1 and .kibana_2 indices and .kibana is an alias to .kibana_2 in dev from the start.
    $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/indices | grep .kibana
    green open .kibana_1             5YTNzmFPRVe8cxl1kfprVg 1 1 1 0   7.2kb   3.6kb
    green open .kibana_2             CHk5z7uJR_GuSMhg3NZAuA 1 1 3 0  24.6kb  13.6kb
    $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
    .kibana   .kibana_2   - - -
  2. Curator successfully closes, restores and opens the .kibana_1 index from a snapshot made in prod to dev (note that prod only has one .kibana_n index and it is .kibana_1 aliased as .kibana):
    2020-03-20 04:45:14,773 INFO       curator.actions.snapshot           do_action:1781 Restoring indices "['.kibana_1']" from snapshot: kibana-20200320010015
    2020-03-20 04:45:14,907 DEBUG              curator.utils            wait_for_it:1765 Elapsed time: 0 seconds
    2020-03-20 04:45:14,911 INFO               curator.utils          restore_check:1608 _recovery returned an empty response. Trying again.
    2020-03-20 04:45:14,911 DEBUG              curator.utils            wait_for_it:1768 Response: False
    2020-03-20 04:45:14,911 DEBUG              curator.utils            wait_for_it:1788 Action "restore" not yet complete, 0 total seconds elapsed. Waiting 10 seconds before checking again.
    2020-03-20 04:45:24,921 DEBUG              curator.utils            wait_for_it:1765 Elapsed time: 10 seconds
    2020-03-20 04:45:24,927 INFO               curator.utils          restore_check:1611 Provided indices: ['.kibana_1']
    2020-03-20 04:45:24,927 INFO               curator.utils          restore_check:1612 Found indices: ['.kibana_1']
    2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1768 Response: True
    2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1773 Action "restore" finished executing (may or may not have been successful)
    2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1791 Result: True
  3. After the restore, I end up with the .kibana alias pointing to both .kibana_2 and .kibana_1 indices. So it is adding .kibana_1 to an existing alias .kibana which already points to .kibana_2.
    $ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
    .kibana   .kibana_2   - - -
    .kibana   .kibana_1   - - -
  4. Now when I kubectl port-forward kibana and go to https://localhost:5601, I get:
    // 20200320010704
    // https://localhost:5601/app/kibana#/management/kibana/index?_g=()
      "statusCode": 400,
      "error": "Bad Request",
      "message": "Alias [.kibana] has more than one indices associated with it
    [[.kibana_1, .kibana_2]], can't execute a single index op:
    [illegal_argument_exception] Alias [.kibana] has more than one indices associated with it
    [[.kibana_1, .kibana_2]], can't execute a single index op"
    and at this point my Kibana pod is at a 0/1 READY state.

Things I have tried:

  1. Remove .kibana_1 from .kibana alias:

    curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d'
    { "actions" : [ { "remove" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }'

    This resulted in an empty Kibana (i.e. no indices, saved objects, visualizations, dashboards, etc.).

  2. Remove .kibana_2 from .kibana alias:

    curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d'
    { "actions" : [ { "remove" : { "index" : ".kibana_2", "alias" : ".kibana" } } ] }'

    This resulted in the following Kibana log message:

    {"type":"log","@timestamp":"2020-03-20T05:27:30Z","tags":["spaces","error"],"pid":1,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"}
    {"type":"response","@timestamp":"2020-03-20T05:27:30Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"","userAgent":""},"res":{"statusCode":302,"responseTime":19,"contentLength":9},"message":"GET /app/kibana 302 19ms - 9.0B"}

    saying it is Unable to navigate to space \"default\" and the following error at https://localhost:5601:

    // 20200320012405
    // https://localhost:5601/#/management/kibana/index?_g=()
      "statusCode": 400,
      "error": "Bad Request",
      "message": "[illegal_argument_exception] application privileges must refer to at
    least one resource"

I just want to have Curator restore a prod snapshot of the .kibana_1 index to dev and have it reflected in Kibana.

Any help is appreciated.

in my documention, when I want to restore prod to dev, I have this steps to do :

delete index .kibana in index management
create the alias .kibana :

POST /_aliases
    "actions" : [
        { "add" : { "index" : ".kibana-6", "alias" : ".kibana" } }

then I have my dashboards, saved search...

Hey @Ludovic9, thanks for replying. I am aware that this works from Dev Tools but I am not sure if it's the same case when using Curator.

When I try what you mentioned manually using cURL:

# since I do not have the .kibana metaindex and .kibana is an 
# alias to the .kibana_2 metaindex, I will remove the alias first
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [ { "remove": { "index": ".kibana_2", "alias": ".kibana" } } ] }'

# START NOTE once I remove the .kibana_2 metaindex from the 
# .kibana alias, ES will quickly try to recreate the default 
# .kibana index. So I will need to execute the next 2 cUrl 
# commands before ES recreates that default .kibana index END NOTE

# then I will delete the .kibana_2 metaindex
$ curl -XDELETE -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2
# then I will create the alias .kibana where .kibana_1 is the
# already restored metaindex (a snapshot of production)
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [ { "add": { "index": ".kibana_1", "alias": ".kibana" } } ] }'

Now, when I kubectl port-forward and go to https://localhost:5601, I keep getting the [illegal_argument_exception] application privileges must refer to at least one resource error due to missing the default space document. What else do I need to do to create this default space document when using Curator? I am currently just doing actions close, restore, and open in order.

Looks like I am facing this very issue that has been open since December of 2018.

So by missing this default space document, I am only able to access Discover, Visualize, Dashboard, Dev Tools, and Management.

The suggested solution is to stop the Kibana server and restart it and I don't like that idea since a restart will create downtime for the users.

I have tried manually creating this space:default document but got an error:

$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_1/_doc/space:default -H 'Content-Type: application/json' -d'
{ "space": { "name": "Default",
             "description": "This is your default space!",
             "color": "#00bfb3",
             "disabledFeatures": [],
             "_reserved": true },
  "type": "space",
  "references": [],
  "migrationVersion": { "space": "6.6.0" } }'
{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"},"status":400}

Is there a workaround?

EDIT: I realized I am trying to restore a snapshot (from a cluster without XPACK enabled) to a cluster with XPACK enabled and this restored snapshot of .kibana_n index does not have this space:default document (since this is an XPACK feature). What can be done in this case?

EDIT 2: I just tried to restore a snapshot (that I just took from the same cluster with XPACK enabled) and got the same error "[illegal_argument_exception] application privileges must refer to at least one resource". The snapshot is missing the space:default document.

The original .kibana_2 index has it defined:

$ curl -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2/_search?pretty
  "_index" : ".kibana_2",
  "_type" : "doc",
  "_id" : "space:default",
  "_score" : 1.0,
  "_source" : {
    "space" : {
      "name" : "Default",
      "description" : "This is your default space!",
      "color" : "#00bfb3",
      "_reserved" : true
    "type" : "space",
    "updated_at" : "2020-03-23T21:39:56.291Z"

but the restored .kibana_1 index does not have this doc. Why? This is preventing Curator from restoring.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.