# Elasticsearch and Curator restore action for .kibana/.kibana\_n indices

**URL:** <https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350>\
**Category:** Kibana\
**Tags:** curator\
**Created:** [March 20, 2020, 5:37am UTC](https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350 "2020-03-20T05:37:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 20, 2020, 5:37am UTC](https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350/1 "2020-03-20T05:37:08Z")

</div>

#### Docker image versions:

Elasticsearch `6.8.2`  
Kibana `6.8.2`

Before I ask the Curator question, I have a few questions about Elasticsearch and `.kibana`/`.kibana_n` indices migration when Kibana starts.

**Note:** Before the Kibana pod restarted, I deleted all indices starting with `.kibana`.

See the following log messages from the Kibana pod:

```auto
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:07Z","tags":["info","migrations"],"pid":1,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-03-20T04:33:08Z","tags":["info","migrations"],"pid":1,"message":"Finished in 972ms."}

```

1. Why is it `Creating index .kibana_2`?
2. Why is it `Reindexing .kibana to .kibana_1`?
3. Why is it `Migrating .kibana_1 saved objects to .kibana_2`?
4. Why is it `Pointing alias .kibana to .kibana_2`?

I ask these questions because (like [this person](https://discuss.elastic.co/t/restore-kibana-index/162806)) I am trying to restore the `.kibana_1` index (i.e. saved objects, visualizations, dashboards, etc.) from a snapshot of a prod Elasticsearch to a dev Elasticsearch. But in my case, I am using Curator.

#### What is happening:

1. When the dev Elasticsearch starts, it inevitably runs this `migrations` thing (see above logs) and I end up with `.kibana_1` and `.kibana_2` indices and `.kibana` is an alias to `.kibana_2` in dev from the start.

```auto
$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/indices | grep .kibana
green open .kibana_1 5YTNzmFPRVe8cxl1kfprVg 1 1 1 0 7.2kb 3.6kb
green open .kibana_2 CHk5z7uJR_GuSMhg3NZAuA 1 1 3 0 24.6kb 13.6kb
$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
.kibana .kibana_2 - - -

```

2. Curator successfully closes, restores and opens the `.kibana_1` index from a snapshot made in prod to dev (note that prod only has one `.kibana_n` index and it is `.kibana_1` aliased as `.kibana`):

```auto
2020-03-20 04:45:14,773 INFO curator.actions.snapshot do_action:1781 Restoring indices "['.kibana_1']" from snapshot: kibana-20200320010015
2020-03-20 04:45:14,907 DEBUG curator.utils wait_for_it:1765 Elapsed time: 0 seconds
2020-03-20 04:45:14,911 INFO curator.utils restore_check:1608 _recovery returned an empty response. Trying again.
2020-03-20 04:45:14,911 DEBUG curator.utils wait_for_it:1768 Response: False
2020-03-20 04:45:14,911 DEBUG curator.utils wait_for_it:1788 Action "restore" not yet complete, 0 total seconds elapsed. Waiting 10 seconds before checking again.
2020-03-20 04:45:24,921 DEBUG curator.utils wait_for_it:1765 Elapsed time: 10 seconds
2020-03-20 04:45:24,927 INFO curator.utils restore_check:1611 Provided indices: ['.kibana_1']
2020-03-20 04:45:24,927 INFO curator.utils restore_check:1612 Found indices: ['.kibana_1']
2020-03-20 04:45:24,927 DEBUG curator.utils wait_for_it:1768 Response: True
2020-03-20 04:45:24,927 DEBUG curator.utils wait_for_it:1773 Action "restore" finished executing (may or may not have been successful)
2020-03-20 04:45:24,927 DEBUG curator.utils wait_for_it:1791 Result: True

```

3. After the restore, I end up with the `.kibana` alias pointing to both `.kibana_2` and `.kibana_1` indices. So it is adding `.kibana_1` to an existing alias `.kibana` which already points to `.kibana_2`.

```auto
$ curl -k -u elastic:elastic https://elasticsearch-master:9200/_cat/aliases
.kibana .kibana_2 - - -
.kibana .kibana_1 - - -

```

4. Now when I `kubectl port-forward kibana` and go to `https://localhost:5601`, I get:

```auto
// 20200320010704
// https://localhost:5601/app/kibana#/management/kibana/index?_g=()

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Alias [.kibana] has more than one indices associated with it
[[.kibana_1, .kibana_2]], can't execute a single index op:
[illegal_argument_exception] Alias [.kibana] has more than one indices associated with it
[[.kibana_1, .kibana_2]], can't execute a single index op"
}

```

and at this point my Kibana pod is at a `0/1 READY` state.

* * *

#### Things I have tried:

1. Remove `.kibana_1` from `.kibana` alias:

2. Remove `.kibana_2` from `.kibana` alias:

I just want to have Curator restore a prod snapshot of the `.kibana_1` index to dev and have it reflected in Kibana.

Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![Ludovic9](https://avatars.discourse-cdn.com/v4/letter/l/977dab/32.png) [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\
**Post date:** [March 20, 2020, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350/2 "2020-03-20T08:25:01Z")

</div>

hello  
in my documention, when I want to restore prod to dev, I have this steps to do :

delete index .kibana in index management  
create the alias .kibana :

```
POST /_aliases
{
    "actions" : [
        { "add" : { "index" : ".kibana-6", "alias" : ".kibana" } }
    ]
}

```

then I have my dashboards, saved search...

---

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 24, 2020, 3:13am UTC](https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350/3 "2020-03-24T03:13:05Z")

</div>

Hey @Ludovic9, thanks for replying. I am aware that this works from Dev Tools but I am not sure if it's the same case when using Curator.

When I try what you mentioned manually using cURL:

1. 

> [@Ludovic9](#):
>
> delete index .kibana in index management

```auto
# since I do not have the .kibana metaindex and .kibana is an 
# alias to the .kibana_2 metaindex, I will remove the alias first
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [{ "remove": { "index": ".kibana_2", "alias": ".kibana" } }] }'
{"acknowledged":true}

# START NOTE once I remove the .kibana_2 metaindex from the 
# .kibana alias, ES will quickly try to recreate the default 
# .kibana index. So I will need to execute the next 2 cUrl 
# commands before ES recreates that default .kibana index END NOTE

# then I will delete the .kibana_2 metaindex
$ curl -XDELETE -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2
{"acknowledged":true}

```

1. 

> [@Ludovic9](#):
>
> create the alias .kibana

```auto
# then I will create the alias .kibana where .kibana_1 is the
# already restored metaindex (a snapshot of production)
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/_aliases -H "Content-Type: application/json" -d'
{ "actions": [{ "add": { "index": ".kibana_1", "alias": ".kibana" } }] }'

```

Now, when I `kubectl port-forward` and go to [https://localhost:5601](https://localhost:5601), I keep getting the `[illegal_argument_exception] application privileges must refer to at least one resource` error due to missing the `default space document`. What else do I need to do to create this `default space document` when using Curator? I am currently just doing actions `close`, `restore`, and `open` in order.

---

<div class="post-metadata">

**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\
**Post date:** [March 24, 2020, 3:58am UTC](https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350/4 "2020-03-24T03:58:00Z")

</div>

Looks like I am facing [this very issue](https://github.com/elastic/kibana/issues/26845) that has been open since December of 2018.

So by missing this `default space document`, I am only able to access `Discover`, `Visualize`, `Dashboard`, `Dev Tools`, and `Management`.

The suggested solution is to stop the Kibana server and restart it and I don't like that idea since a restart will create downtime for the users.

I have tried manually creating this `space:default document` but got an error:

```auto
$ curl -XPOST -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_1/_doc/space:default -H 'Content-Type: application/json' -d'
{ "space": { "name": "Default",
             "description": "This is your default space!",
             "color": "#00bfb3",
             "disabledFeatures": [],
             "_reserved": true },
  "type": "space",
  "references": [],
  "migrationVersion": { "space": "6.6.0" } }'
{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [space] within [doc] is not allowed"},"status":400}

```

Is there a workaround?

EDIT: I realized I am trying to restore a snapshot (from a cluster without `XPACK` enabled) to a cluster with `XPACK` enabled and this restored snapshot of `.kibana_n` index does not have this `space:default document` (since this is an `XPACK` feature). What can be done in this case?

EDIT 2: I just tried to restore a snapshot (that I just took from the same cluster with `XPACK` enabled) and got the same error `"[illegal_argument_exception] application privileges must refer to at least one resource"`. The snapshot is missing the `space:default document`.

The original `.kibana_2` index has it defined:

```auto
$ curl -k -u elastic:elastic https://elasticsearch-master:9200/.kibana_2/_search?pretty
...
{
  "_index" : ".kibana_2",
  "_type" : "doc",
  "_id" : "space:default",
  "_score" : 1.0,
  "_source" : {
    "space" : {
      "name" : "Default",
      "description" : "This is your default space!",
      "color" : "#00bfb3",
      "_reserved" : true
    },
    "type" : "space",
    "updated_at" : "2020-03-23T21:39:56.291Z"
  }
}
...

```

but the restored `.kibana_1` index does not have this doc. Why? This is preventing Curator from restoring.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 21, 2020, 3:58am UTC](https://discuss.elastic.co/t/elasticsearch-and-curator-restore-action-for-kibana-kibana-n-indices/224350/5 "2020-04-21T03:58:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.