Backup indices and data stored in ELK

HI guys, I am rather new to ELK (7.16). So far I have registered a repository and can create now snapshots. I have used the following API command:

PUT /_snapshot/BackUpELKData/snapshot_2?wait_for_completion=true
{

"indices": "-.ds-ilm-history-5-2022.08.23-000001",

"ignore_unavailable": true,

"include_global_state": false

}

The index I excluded caused problems when restoring from the snapshot.
My goal however is to create a backup of the used indices and the data files.

When I created the snapshot with some indices and data available for Discovery and Dashboards and deleted the indices after creation, I tried to restore them with the snapshot.
The process of restoring was working with no problems. I received no errors.
The indices however where still missing so I could not access the data. Neither in Discovery nor in Dashboards.
I have read here about snapshots and I am not sure if I have done everything correctly.
Can someone help me and explain how I can create a full backup of all data and indices which I can restore at any point?

All the best,
Clonky

If I want to restore the snapshot, I use the API command

POST /_snapshot/BackUpELKData/snapshot/_restore
{
  "indices": "pq_reports"
}

However, I get the error message

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [pq_reports]",
        "index_uuid" : "_na_",
        "index" : "pq_reports"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [pq_reports]",
    "index_uuid" : "_na_",
    "index" : "pq_reports"
  },
  "status" : 404
}

This message comes when I want to restore from a snapshot when the indices were still open and when I closed the indices before creating the snapshot

What is the output from Get snapshot API | Elasticsearch Guide [8.4] | Elastic?

Good Morning, the output of

GET /_snapshot/my_repository/my_snapshot

is

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_missing_exception",
        "reason" : "[my_repository] missing"
      }
    ],
    "type" : "repository_missing_exception",
    "reason" : "[my_repository] missing"
  },
  "status" : 404
}

The output of the backup API command is

{
  "snapshot" : {
    "snapshot" : "snapshot_1",
    "uuid" : "eoq0PodYTnGz94f7GfMgXw",
    "repository" : "BackUpELKData",
    "version_id" : 7160399,
    "version" : "7.16.3",
    "indices" : [ ],
    "data_streams" : [ ],
    "include_global_state" : false,
    "state" : "SUCCESS",
    "start_time" : "2022-08-29T06:25:25.585Z",
    "start_time_in_millis" : 1661754325585,
    "end_time" : "2022-08-29T06:25:25.585Z",
    "end_time_in_millis" : 1661754325585,
    "duration_in_millis" : 0,
    "failures" : [ ],
    "shards" : {
      "total" : 0,
      "failed" : 0,
      "successful" : 0
    },
    "feature_states" : [ ]
  }
}

You are not naming any indices to be added to the snapshot.

Please show the full snapshot API call... That provided that result.

Hello, the whole API call that provided the output is

PUT /_snapshot/BackUpELKData/snapshot_1?wait_for_completion=true
{

"indices": "-.ds-ilm-history-5-2022.08.23-000001",

"ignore_unavailable": true,

"include_global_state": false

}

I thought that with this I just excluded the one indice but included all the other ones. Especially the ones I created for my data

All the best

After some more research with the different APIs, I found the command

PUT /_snapshot/BackUpELKData/my_snapshot

which created a snapshot of all the indices. See the result of

GET /_snapshot/BackUpELKData/my_snapshot
{
  "snapshots" : [
    {
      "snapshot" : "my_snapshot",
      "uuid" : "hUq_mzefT4af4azpFgqQ_Q",
      "repository" : "BackUpELKData",
      "version_id" : 7160399,
      "version" : "7.16.3",
      "indices" : [
        "test-results-abcd",
        "test-results-efgh",
        "ijk_results",
        ".ds-.logs-deprecation.elasticsearch-default-2022.08.23-000001",
        "lmno_logging",
        ".async-search",
        ".kibana_task_manager_7.15.2_001",
        ".ds-.slm-history-5-2022.08.25-000001",
        ".ds-ilm-history-5-2022.08.23-000001",
        "pq_reports",
        ".apm-agent-configuration",
        ".ds-ilm-history-5-2022.08.23-000001xxxx",
        ".kibana-event-log-7.15.2-000001",
        "flashspeed",
        ".apm-custom-link",
        ".kibana_7.15.2_001"
      ],
      "data_streams" : [
        "ilm-history-5",
        ".logs-deprecation.elasticsearch-default",
        ".slm-history-5"
      ],
      "include_global_state" : true,
      "state" : "SUCCESS",
      "start_time" : "2022-08-29T12:45:33.799Z",
      "start_time_in_millis" : 1661777133799,
      "end_time" : "2022-08-29T12:45:34.612Z",
      "end_time_in_millis" : 1661777134612,
      "duration_in_millis" : 813,
      "failures" : [ ],
      "shards" : {
        "total" : 16,
        "failed" : 0,
        "successful" : 16
      },
      "feature_states" : [
        {
          "feature_name" : "async_search",
          "indices" : [
            ".async-search"
          ]
        },
        {
          "feature_name" : "kibana",
          "indices" : [
            ".apm-custom-link",
            ".apm-agent-configuration",
            ".kibana_7.15.2_001",
            ".kibana_task_manager_7.15.2_001"
          ]
        }
      ]
    }
  ],
  "total" : 1,
  "remaining" : 0
}

With this, I can now restore any index via the command

POST _snapshot/BackUpELKData/my_snapshot/_restore
{
  "indices": "lmno_logging"
}

But how can I exclude indices from the backup? Lets say, I do not want to backup the index "lmno_logging". How should I change the command from above which created the snapshot?

1 Like

No that just says ingore that one... and give no other instructions... so no indices are backed up I think you were trying to

"indices": "*,-.ds-ilm-history-5-2022.08.23-000001",

Which would say all but that index.... Once you put an index in there you need to be precise.

Thanks a lot! The snipped is working now.

Thanks for your help!