Snapshot Policy Change not Deleting Old Snapshots

Elastic Stack Elasticsearch
1

Elastic 7.6.2

We changed our snapshot policy to only keep max 5 snapshots (down from 14). Nothing happened. We ran the snapshot (from Kibana) and a new snapshot was generated. Nothing was deleted and no errors were shown/logged. We expected the number of snapshots to reduce to the last 5.

At what point do the snapshots get cleaned up/deleted and how can we trigger this process.

[2022-03-24T08:12:26,054][INFO ][o.e.x.s.a.TransportPutSnapshotLifecycleAction] [ecom-db01] updating existing snapshot lifecycle [daily-parts]
[2022-03-24T08:12:26,122][INFO ][o.e.x.s.SnapshotLifecycleService] [ecom-db01] rescheduling updated snapshot lifecycle job [daily-parts-2]
[2022-03-24T08:12:51,843][INFO ][o.e.x.s.a.TransportPutSnapshotLifecycleAction] [ecom-db01] updating existing snapshot lifecycle [daily-machines]
[2022-03-24T08:12:51,888][INFO ][o.e.x.s.SnapshotLifecycleService] [ecom-db01] rescheduling updated snapshot lifecycle job [daily-machines-3]
[2022-03-24T08:13:16,767][INFO ][o.e.x.s.SnapshotLifecycleTask] [ecom-db01] snapshot lifecycle policy [daily-parts] issuing create snapshot [parts-2022.03.24-ce-0yymoqeivplai8q5bta]
[2022-03-24T08:13:16,794][INFO ][o.e.s.SnapshotsService   ] [ecom-db01] snapshot [parts:parts-2022.03.24-ce-0yymoqeivplai8q5bta/MShAVQHUTTOrAoimMHqaQA] started
[2022-03-24T08:13:23,545][INFO ][o.e.s.SnapshotsService   ] [ecom-db01] snapshot [parts:parts-2022.03.24-ce-0yymoqeivplai8q5bta/MShAVQHUTTOrAoimMHqaQA] completed with state [SUCCESS]
[2022-03-24T08:47:43,762][INFO ][o.e.m.j.JvmGcMonitorService] [ecom-db01] [gc][3403612] overhead, spent [264ms] collecting in the last [1s]
-03-24T08:13:23,545][INFO ][o.e.s.SnapshotsService   ] [ecom-db01] snapshot [parts:parts-2022.03.24-ce-0yymoqeivplai8q5bta/MShAVQHUTTOrAoimMHqaQA] completed with state [SUCCESS]

GET _cat/snapshots/parts
parts-20211126                          SUCCESS 1637923540 10:45:40 1637923904 10:51:44    6m 2 2 0 2
parts-20220310                          SUCCESS 1646924881 15:08:01 1646925110 15:11:50  3.8m 2 2 0 2
machines-20220310                       SUCCESS 1646928587 16:09:47 1646928611 16:10:11 24.4s 1 1 0 1
parts-20220315                          SUCCESS 1647325686 06:28:06 1647325888 06:31:28  3.3m 1 1 0 1
parts-2022.03.16-fl_okjejt_q1ddn0izeygw SUCCESS 1647394206 01:30:06 1647394294 01:31:34  1.4m 1 1 0 1
parts-2022.03.17-a9ftqwfrspweq46rzbkekq SUCCESS 1647480611 01:30:11 1647480688 01:31:28  1.2m 1 1 0 1
parts-2022.03.18-s2l7h4cpqneirui0r6z60w SUCCESS 1647567008 01:30:08 1647567104 01:31:44  1.5m 1 1 0 1
parts-2022.03.19-gaj0coqtrsc8gaotjduomg SUCCESS 1647653409 01:30:09 1647653440 01:30:40 31.2s 1 1 0 1
parts-2022.03.20-vtvw8eryq92vqh5yd_v4ig SUCCESS 1647739808 01:30:08 1647739885 01:31:25  1.2m 1 1 0 1
parts-2022.03.21-psbqpmydsfetgpgktyhzwq SUCCESS 1647826208 01:30:08 1647826228 01:30:28 19.9s 1 1 0 1
parts-2022.03.22-gzlvjpuwtjkxsimjemsflg SUCCESS 1647912608 01:30:08 1647912640 01:30:40 31.2s 1 1 0 1
parts-2022.03.23-ylhr1efkr92khk8uuzo4ag SUCCESS 1647999009 01:30:09 1647999053 01:30:53 44.2s 1 1 0 1
parts-2022.03.24-oldbyszzshgefwiybji69w SUCCESS 1648085409 01:30:09 1648085557 01:32:37  2.4m 1 1 0 1
parts-2022.03.24-ce-0yymoqeivplai8q5bta SUCCESS 1648105996 07:13:16 1648106003 07:13:23  6.8s 1 1 0 1

GET _slm/stats
{
  "retention_runs" : 9,
  "retention_failed" : 0,
  "retention_timed_out" : 0,
  "retention_deletion_time" : "0s",
  "retention_deletion_time_millis" : 0,
  "total_snapshots_taken" : 19,
  "total_snapshots_failed" : 0,
  "total_snapshots_deleted" : 0,
  "total_snapshot_deletion_failures" : 0,
  "policy_stats" : [
    {
      "policy" : "daily-machines",
      "snapshots_taken" : 9,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 0,
      "snapshot_deletion_failures" : 0
    },
    {
      "policy" : "daily-parts",
      "snapshots_taken" : 10,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 0,
      "snapshot_deletion_failures" : 0
    }
  ]
}
2

use this API to do it:

1 Like
3

and you can change the policy by this:

PUT _cluster/settings
{
    "persistent": {
        "slm.retention_schedule": "0 0 21 * * ?",
        "slm.retention_duration":"1h"
    }
}
4

Thanks! Does this mean the SLM Retention Policy never runs until I PUT _cluster/settings? When I GET _cluster/settings I see no SLM entries.

5

Oh! I see there IS a default setting for every day at 01:30 AM

image
image1507×285 27.8 KB

6

It has default values use this to get：

GET _cluster/settings?include_defaults
2 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.