Backup indices and data stored in ELK

Elastic Stack Elasticsearch
1

HI guys, I am rather new to ELK (7.16). So far I have registered a repository and can create now snapshots. I have used the following API command:

PUT /_snapshot/BackUpELKData/snapshot_2?wait_for_completion=true
{

"indices": "-.ds-ilm-history-5-2022.08.23-000001",

"ignore_unavailable": true,

"include_global_state": false

}

The index I excluded caused problems when restoring from the snapshot.
My goal however is to create a backup of the used indices and the data files.

When I created the snapshot with some indices and data available for Discovery and Dashboards and deleted the indices after creation, I tried to restore them with the snapshot.
The process of restoring was working with no problems. I received no errors.
The indices however where still missing so I could not access the data. Neither in Discovery nor in Dashboards.
I have read here about snapshots and I am not sure if I have done everything correctly.
Can someone help me and explain how I can create a full backup of all data and indices which I can restore at any point?

All the best,
Clonky

2

If I want to restore the snapshot, I use the API command

POST /_snapshot/BackUpELKData/snapshot/_restore
{
  "indices": "pq_reports"
}

However, I get the error message

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [pq_reports]",
        "index_uuid" : "_na_",
        "index" : "pq_reports"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [pq_reports]",
    "index_uuid" : "_na_",
    "index" : "pq_reports"
  },
  "status" : 404
}

This message comes when I want to restore from a snapshot when the indices were still open and when I closed the indices before creating the snapshot

3

What is the output from Get snapshot API | Elasticsearch Guide [8.4] | Elastic?

4

Good Morning, the output of

GET /_snapshot/my_repository/my_snapshot

is

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_missing_exception",
        "reason" : "[my_repository] missing"
      }
    ],
    "type" : "repository_missing_exception",
    "reason" : "[my_repository] missing"
  },
  "status" : 404
}

The output of the backup API command is

{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "uuid" : "eoq0PodYTnGz94f7GfMgXw",
    "repository" : "BackUpELKData",
    "version_id" : 7160399,
    "version" : "7.16.3",
    "indices" : [ ],
    "data_streams" : [ ],
    "include_global_state" : false,
    "state" : "SUCCESS",
    "start_time" : "2022-08-29T06:25:25.585Z",
    "start_time_in_millis" : 1661754325585,
    "end_time" : "2022-08-29T06:25:25.585Z",
    "end_time_in_millis" : 1661754325585,
    "duration_in_millis" : 0,
    "failures" : [ ],
    "shards" : {
      "total" : 0,
      "failed" : 0,
      "successful" : 0
    },
    "feature_states" : [ ]
  }
}
5

You are not naming any indices to be added to the snapshot.

Please show the full snapshot API call... That provided that result.

6

Hello, the whole API call that provided the output is

PUT /_snapshot/BackUpELKData/snapshot_1?wait_for_completion=true
{

"indices": "-.ds-ilm-history-5-2022.08.23-000001",

"ignore_unavailable": true,

"include_global_state": false

}

I thought that with this I just excluded the one indice but included all the other ones. Especially the ones I created for my data

All the best

7

After some more research with the different APIs, I found the command

PUT /_snapshot/BackUpELKData/my_snapshot

which created a snapshot of all the indices. See the result of

GET /_snapshot/BackUpELKData/my_snapshot

{
  "snapshots" : [
    {
      "snapshot" : "my_snapshot",
      "uuid" : "hUq_mzefT4af4azpFgqQ_Q",
      "repository" : "BackUpELKData",
      "version_id" : 7160399,
      "version" : "7.16.3",
      "indices" : [
        "test-results-abcd",
        "test-results-efgh",
        "ijk_results",
        ".ds-.logs-deprecation.elasticsearch-default-2022.08.23-000001",
        "lmno_logging",
        ".async-search",
        ".kibana_task_manager_7.15.2_001",
        ".ds-.slm-history-5-2022.08.25-000001",
        ".ds-ilm-history-5-2022.08.23-000001",
        "pq_reports",
        ".apm-agent-configuration",
        ".ds-ilm-history-5-2022.08.23-000001xxxx",
        ".kibana-event-log-7.15.2-000001",
        "flashspeed",
        ".apm-custom-link",
        ".kibana_7.15.2_001"
      ],
      "data_streams" : [
        "ilm-history-5",
        ".logs-deprecation.elasticsearch-default",
        ".slm-history-5"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2022-08-29T12:45:33.799Z",
      "start_time_in_millis" : 1661777133799,
      "end_time" : "2022-08-29T12:45:34.612Z",
      "end_time_in_millis" : 1661777134612,
      "duration_in_millis" : 813,
      "failures" : [ ],
      "shards" : {
        "total" : 16,
        "failed" : 0,
        "successful" : 16
      },
      "feature_states" : [
        {
          "feature_name" : "async_search",
          "indices" : [
            ".async-search"
          ]
        },
        {
          "feature_name" : "kibana",
          "indices" : [
            ".apm-custom-link",
            ".apm-agent-configuration",
            ".kibana_7.15.2_001",
            ".kibana_task_manager_7.15.2_001"
          ]
        }
      ]
    }
  ],
  "total" : 1,
  "remaining" : 0
}

With this, I can now restore any index via the command

POST _snapshot/BackUpELKData/my_snapshot/_restore
{
  "indices": "lmno_logging"
}

But how can I exclude indices from the backup? Lets say, I do not want to backup the index "lmno_logging". How should I change the command from above which created the snapshot?

1 Like
8

No that just says ingore that one... and give no other instructions... so no indices are backed up I think you were trying to

"indices": "*,-.ds-ilm-history-5-2022.08.23-000001",

Which would say all but that index.... Once you put an index in there you need to be precise.

9

Thanks a lot! The snipped is working now.

Thanks for your help!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.