Elasticsearch Curator snapshot/restore with multiple .kibana/.kibana_N indices

Elastic Stack Elasticsearch
1

I am running two separate instances (x and y) of ES.

I am treating y as the "source of truth" for saved objects. So x will snapshot to x's s3 bucket (as a backup plan if y goes stale) but always restore from y's s3 bucket using Curator.

y will snapshot to y's s3 bucket and never restore (at least not with Curator).

For whatever reason, each of my ES instances has both .kibana and .kibana_1 indices and .kibana is an alias for .kibana_1 which complicates things for the restore.

I am running Curator (as a k8s CronJob) to snapshot like the following:

actions:
  1:
    action: snapshot
    description: Creates a snapshot of .kibana indices
    options:
      ignore_empty_list: True
      repository: backup-x # y is backing up to backup-y
      name: 'kibana-%Y%m%d%H%M%S'
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: pattern
      kind: prefix
      value: '.kibana'

And for x, I have set up the s3 repository to point to the y's snapshot s3 bucket and am running Curator (as a k8s CronJob) to restore like the following:

actions:
  1:
    action: restore
    description: Restores the .kibana indices from the latest snapshot with state SUCCESS
    options:
      ignore_empty_list: True
      repository: backup-y
      name:
      indices: ['.kibana_1'] # I have tried ['.kibana'] and ['.kibana', '.kibana_1'] as well
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
      - filtertype: state
        state: SUCCESS

All I want is to overwrite the Kibana saved objects of x with the Kibana saved objects of y's latest snapshot but the restore keeps failing with the following error:

kubectl logs -n <my-namespace> elk-elasticsearch-curator-restore-1583451000-9b98d
2020-03-05 23:31:23,843 INFO      Preparing Action ID: 1, "restore"
2020-03-05 23:31:23,851 INFO      Trying Action ID: 1, "restore": Restores the .kibana indices from the latest snapshot with state SUCCESS
2020-03-05 23:31:25,512 INFO      Restoring indices "['.kibana_1']" from snapshot: kibana-20200305010010
2020-03-05 23:31:25,580 ERROR     Failed to complete action: restore.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.  Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: TransportError(500, 'snapshot_restore_exception', '[backup-y:kibana-20200305010010/ZeYoq_gCTOuLt5kPFFHqOQ] cannot restore index [.kibana_1] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name')

What am I missing? Do I need to delete the indices before restore? Or do I need to close the indices before restore? I just assumed Curator would do this seamlessly.

2

Indeed, you cannot restore to an existing index automatically. You have a few options:

  1. Close the destination indices first. You can restore to an existing index if it is closed (don't forget to re-open afterwards).
  2. Delete the destination indices first.
  3. Since .kibana is an alias, you could restore to .kibana_1-restored and point the .kibana alias to it after restoration. This uses rename/replace functionality in the Elasticsearch API.
3

I went with the first option and was able to get it working seamlessly.

Thank you and have a wonderful weekend! @theuntergeek

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.