How to create a snapshot only for previous day index using curator

Elastic Stack Elasticsearch
1

HI ,

i want to create a snapshot daily of my previous day index but the snapshot indices are incremental.

i need help to solve it ,is there any way we can configure to snapshot only previous day indices or delete the unnecessary indices inside a snapshot using curator ?

i want to snapshot daily but only the previous day index and there should not be any repetitive indices in a snapshot.

2

Why do you want to do this? Snapshots are incremental, but not at the data level. They are incremental at the segment level. What this means is that if no segments from the index you want to snapshot are on the repository, all of the segments from the index will be copied over. It's not incremental in the sense that a tar backup will only copy files with a newer mtime.

If you want to have only a single index in a snapshot, you only need a separate snapshot name every day with only one index in it, which does not actually affect the way segments are copied or compared in any way.

3

@theuntergeek thanks for the reply ,

my problem is the following

input in kibana devtools console :

GET /_snapshot/backup/_all

output :

{
"snapshots" : [
{
"snapshot" : "logstash_backup-2019-06-21",
"uuid" : "BJjwyCFmRZCuDxKJwemn2w",
"version_id" : 6080099,
"version" : "6.8.0",
"indices" : [
"logstash-2019.06.05",
"logstash-2019.06.20",
"logstash-2019.06.18",
"logstash-2019.06.19"
],
"include_global_state" : true,
"state" : "SUCCESS",
"start_time" : "2019-06-21T07:43:21.068Z",
"start_time_in_millis" : 1561103001068,
"end_time" : "2019-06-21T07:43:23.058Z",
"end_time_in_millis" : 1561103003058,
"duration_in_millis" : 1990,
"failures" : ,
"shards" : {
"total" : 20,
"failed" : 0,
"successful" : 20
}
},
{
"snapshot" : "logstash_backup-2019-06-22",
"uuid" : "tqV08lv4TieiMhHFEFd-yw",
"version_id" : 6080099,
"version" : "6.8.0",
"indices" : [
"logstash-2019.06.05",
"logstash-2019.06.20",
"logstash-2019.06.18",
"logstash-2019.06.19",
"logstash-2019.06.21"
],
"include_global_state" : true,
"state" : "SUCCESS",
"start_time" : "2019-06-22T18:30:04.867Z",
"start_time_in_millis" : 1561228204867,
"end_time" : "2019-06-22T18:30:08.759Z",
"end_time_in_millis" : 1561228208759,
"duration_in_millis" : 3892,
"failures" : [ ],
"shards" : {
"total" : 20,
"failed" : 0,
"successful" : 20
}
}
]
}

so if you observe closely the snapshot : logstash_backup-2019-06-22 snapshotted all the previous days , i already had the backup of it and i dont want to have a snapshot for same index twice .

for the next day i want to backup or snapshot the index for which i didn't have snapshot.

example for the way i want to configure

logstash_backup-2019-06-21
"logstash-2019.06.05",
"logstash-2019.06.20",
"logstash-2019.06.18",
"logstash-2019.06.19"

logstash_backup-2019-06-22
"logstash-2019.06.21"

since i already had snapshot of all previous indices i want to create a snapshot for only the index i didn't snapshot till now.

is there any solution for that ?

4

Yes, but if you understand how snapshots work—at the segment level—it means that only segments that have changed will be actually copied. While you see that an index has been snapshotted, it does not necessarily mean that any new data (segments) have been copied over. If no new segments have been copied, then all that happens is a pointer to segments already on the repository are added to that snapshot. To be clear, I will restate this: Even though you see multiple indices in the snapshot, it does not mean that data (segments) from those indices were re-copied to the repository. Only if new segments were created in those indices would anything be copied to the repository.

With that understanding, there is generally no compelling reason to manually force one index per snapshot. However, if you still want to do this, there are still ways to do so.

- filtertype: age
  source: name
  direction: older
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 1
- filtertype: age
  source: name
  direction: younger
  timestring: '%Y.%m.%d'
  unit: days
  unit_count: 2

This example filter list will include indices older than 1 days, but also only include indices newer than 2 days

5

hi @theuntergeek ,

i am getting the following errors

2019-06-26 12:21:18,472 ERROR Schema error: not a valid value for dictionary value @ data['direction']

2019-06-26 12:21:18,472 ERROR Schema error: Configuration: filter: Location: Action ID "1", action "snapshot", filter #2: {'direction': 'newer', 'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit_count': 2, 'unit': 'days'}: Bad Value: "newer", not a valid value for dictionary value @ data['direction']. Check configuration file.

6

Apologies. I corrected the example to read younger. I didn't verify when I posted. It is younger in the documentation.

7

thanks @theuntergeek , it worked like a charm.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.