Elasticsearch and Curator restore action for .kibana/.kibana_n indices

danksim · March 20, 2020, 5:37am

Docker image versions:

Elasticsearch 6.8.2
Kibana 6.8.2

Before I ask the Curator question, I have a few questions about Elasticsearch and .kibana/.kibana_n indices migration when Kibana starts.

Note: Before the Kibana pod restarted, I deleted all indices starting with .kibana.

See the following log messages from the Kibana pod:

{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Finished in 972ms."}

Why is it Creating index .kibana_2?
Why is it Reindexing .kibana to .kibana_1?
Why is it Migrating .kibana_1 saved objects to .kibana_2?
Why is it Pointing alias .kibana to .kibana_2?

I ask these questions because (like this person) I am trying to restore the .kibana_1 index (i.e. saved objects, visualizations, dashboards, etc.) from a snapshot of a prod Elasticsearch to a dev Elasticsearch. But in my case, I am using Curator.

What is happening:

When the dev Elasticsearch starts, it inevitably runs this migrations thing (see above logs) and I end up with .kibana_1 and .kibana_2 indices and .kibana is an alias to .kibana_2 in dev from the start.

$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/indices | grep .kibana
green open .kibana_1             5YTNzmFPRVe8cxl1kfprVg 1 1 1 0   7.2kb   3.6kb
green open .kibana_2             CHk5z7uJR_GuSMhg3NZAuA 1 1 3 0  24.6kb  13.6kb
$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
.kibana   .kibana_2   - - -

Curator successfully closes, restores and opens the .kibana_1 index from a snapshot made in prod to dev (note that prod only has one .kibana_n index and it is .kibana_1 aliased as .kibana):

2020-03-20 04:45:14,773 INFO       curator.actions.snapshot           do_action:1781 Restoring indices "['.kibana_1']" from snapshot: kibana-20200320010015
2020-03-20 04:45:14,907 DEBUG              curator.utils            wait_for_it:1765 Elapsed time: 0 seconds
2020-03-20 04:45:14,911 INFO               curator.utils          restore_check:1608 _recovery returned an empty response. Trying again.
2020-03-20 04:45:14,911 DEBUG              curator.utils            wait_for_it:1768 Response: False
2020-03-20 04:45:14,911 DEBUG              curator.utils            wait_for_it:1788 Action "restore" not yet complete, 0 total seconds elapsed. Waiting 10 seconds before checking again.
2020-03-20 04:45:24,921 DEBUG              curator.utils            wait_for_it:1765 Elapsed time: 10 seconds
2020-03-20 04:45:24,927 INFO               curator.utils          restore_check:1611 Provided indices: ['.kibana_1']
2020-03-20 04:45:24,927 INFO               curator.utils          restore_check:1612 Found indices: ['.kibana_1']
2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1768 Response: True
2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1773 Action "restore" finished executing (may or may not have been successful)
2020-03-20 04:45:24,927 DEBUG              curator.utils            wait_for_it:1791 Result: True

After the restore, I end up with the .kibana alias pointing to both .kibana_2 and .kibana_1 indices. So it is adding .kibana_1 to an existing alias .kibana which already points to .kibana_2.
```
$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
.kibana   .kibana_2   - - -
.kibana   .kibana_1   - - -
```

Now when I kubectl port-forward kibana and go to https://localhost:5601, I get:

// 20200320010704
// https://localhost:5601/app/kibana#/management/kibana/index?_g=()

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Alias [.kibana] has more than one indices associated with it
[[.kibana_1, .kibana_2]], can't execute a single index op:
[illegal_argument_exception] Alias [.kibana] has more than one indices associated with it
[[.kibana_1, .kibana_2]], can't execute a single index op"
}

and at this point my Kibana pod is at a 0/1 READY state.

Things I have tried:

Remove .kibana_1 from .kibana alias:

curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d'
{ "actions" : [ { "remove" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }'

This resulted in an empty Kibana (i.e. no indices, saved objects, visualizations, dashboards, etc.).

Remove .kibana_2 from .kibana alias:

curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H 'Content-Type: application/json' -d'
{ "actions" : [ { "remove" : { "index" : ".kibana_2", "alias" : ".kibana" } } ] }'

This resulted in the following Kibana log message:

{"type":"log","@timestamp":"2020-03-20T05:27:30Z","tags":["spaces","error"],"pid":1,"message":"Unable to navigate to space \"default\", redirecting to Space Selector. Error: Saved object [space/default] not found"}
{"type":"response","@timestamp":"2020-03-20T05:27:30Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":302,"responseTime":19,"contentLength":9},"message":"GET /app/kibana 302 19ms - 9.0B"}

saying it is Unable to navigate to space \"default\" and the following error at https://localhost:5601:

// 20200320012405
// https://localhost:5601/#/management/kibana/index?_g=()

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "[illegal_argument_exception] application privileges must refer to at
least one resource"
}

I just want to have Curator restore a prod snapshot of the .kibana_1 index to dev and have it reflected in Kibana.

Any help is appreciated.

Ludovic9 · March 20, 2020, 8:25am

hello
in my documention, when I want to restore prod to dev, I have this steps to do :

delete index .kibana in index management
create the alias .kibana :

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : ".kibana-6", "alias" : ".kibana" } }
    ]
}

then I have my dashboards, saved search...

danksim · March 24, 2020, 3:13am

Hey @Ludovic9, thanks for replying. I am aware that this works from Dev Tools but I am not sure if it's the same case when using Curator.

When I try what you mentioned manually using cURL:

# since I do not have the .kibana metaindex and .kibana is an 
# alias to the .kibana_2 metaindex, I will remove the alias first
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [ { "remove": { "index": ".kibana_2", "alias": ".kibana" } } ] }'
{"acknowledged":true}

# START NOTE once I remove the .kibana_2 metaindex from the 
# .kibana alias, ES will quickly try to recreate the default 
# .kibana index. So I will need to execute the next 2 cUrl 
# commands before ES recreates that default .kibana index END NOTE

# then I will delete the .kibana_2 metaindex
$ curl -XDELETE -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2
{"acknowledged":true}

# then I will create the alias .kibana where .kibana_1 is the
# already restored metaindex (a snapshot of production)
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [ { "add": { "index": ".kibana_1", "alias": ".kibana" } } ] }'

Now, when I kubectl port-forward and go to https://localhost:5601, I keep getting the [illegal_argument_exception] application privileges must refer to at least one resource error due to missing the default space document. What else do I need to do to create this default space document when using Curator? I am currently just doing actions close, restore, and open in order.

danksim · March 24, 2020, 3:58am

Looks like I am facing this very issue that has been open since December of 2018.

So by missing this default space document, I am only able to access Discover, Visualize, Dashboard, Dev Tools, and Management.

The suggested solution is to stop the Kibana server and restart it and I don't like that idea since a restart will create downtime for the users.

I have tried manually creating this space:default document but got an error:

$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_1/_doc/space:default -H 'Content-Type: application/json' -d'
{ "space": { "name": "Default",
             "description": "This is your default space!",
             "color": "#00bfb3",
             "disabledFeatures": [],
             "_reserved": true },
  "type": "space",
  "references": [],
  "migrationVersion": { "space": "6.6.0" } }'
{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"},"status":400}

Is there a workaround?

EDIT: I realized I am trying to restore a snapshot (from a cluster without XPACK enabled) to a cluster with XPACK enabled and this restored snapshot of .kibana_n index does not have this space:default document (since this is an XPACK feature). What can be done in this case?

EDIT 2: I just tried to restore a snapshot (that I just took from the same cluster with XPACK enabled) and got the same error "[illegal_argument_exception] application privileges must refer to at least one resource". The snapshot is missing the space:default document.

The original .kibana_2 index has it defined:

$ curl -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2/_search?pretty
...
{
  "_index" : ".kibana_2",
  "_type" : "doc",
  "_id" : "space:default",
  "_score" : 1.0,
  "_source" : {
    "space" : {
      "name" : "Default",
      "description" : "This is your default space!",
      "color" : "#00bfb3",
      "_reserved" : true
    },
    "type" : "space",
    "updated_at" : "2020-03-23T21:39:56.291Z"
  }
}
...

but the restored .kibana_1 index does not have this doc. Why? This is preventing Curator from restoring.

system · April 21, 2020, 3:58am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.