CURATOR: How to DELETE 2 days ago logs

Elastic Stack Elasticsearch
1

Hello! I hope someone can help me

I need to take snapshot only for 1 day per day
and I need to delete logs for 1 day per day

In other words
I want to take snapshots of today a month ago
And delete logs of today a month ago

My CURL DELETE command to delete all logs, looks like this
curl -XDELETE '127.0.0.1:9200/filebeat-*?pretty'
but i need to delete only the logs of a month ago
as also
My snapshot.yml is

actions:
  1:
    action: snapshot
    description: >-
      Snapshot log-production- prefixed indices older than 1 day (based on index
      creation_date) with the default snapshot name pattern of
      'curator-%Y%m%d%H%M%S'.  Wait for the snapshot to complete.  Do not skip
      the repository filesystem access check.  Use the other options to create
      the snapshot.
    options:
      repository: logs_backup

    # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
      name: esdemo-%Y%m%d%H%M%S
      ignore_unavailable: False
      include_global_state: True
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 1

How can I take snapshots a month ago?

Example
Today is March 22
I need to take snapshot of 22 Feb
And then delete the 22 Feb logs

Thanks guys

2

So, why do you only want to snapshot one day at a time? With the way that snapshots work, it is probably not necessary to do so.

Snapshots are at the segment level. What I mean by this is that as new documents/log-lines/etc are indexed, new segments are flushed into each shard. Indices are made up of shards, and shards are made up of segments. When a snapshot is initiated, Elasticsearch freezes the shards for the named indices, preventing merging or alteration of said shards. This state persists until the snapshot is complete. Any document updates are staged in new segments, and then deleted documents are flagged, but not deleted until after the snapshot. This is how Elasticsearch guarantees that a snapshot is a point in time, frozen and immutable.

Part two of the story of segments and snapshots is that Elasticsearch compares the segments stored in the repository to the ones flagged for snapshotting. Only new or changed segments are copied. Any segment that remains unchanged results in the new snapshot adding a pointer to the needed segment. What this means is that so long as there is a pointer to a segment in the repository, Elasticsearch will not delete that segment from the repository.

So, with time-series data, typically, you have no changes to your segments after that day's logging is done. If the segments have been snapshotted once, repeated snapshotting will only result in extra pointers, not extra data being copied. As such, it is not usually a problem for Curator to snapshot yesterday's indices, which won't be further altered. At the end of your snapshot retention period, it can be deleted. Should you use Curator to snapshot all indices older than yesterday, then the same applies. No new data for older indices will be copied, though pointers would be assigned.

So, with that explanation, do you really need a single index per snapshot, per day? Or can you snapshot the next day, and not worry about this "after 30 days" requirement?

I ask, because Curator can easily select an arbitrary month, or n months ago in their entirety. But "today one month ago" is too arbitrary. How would you snapshot the 30th or 31st of February when executing on those days in March? It is far wiser to use a number of days for your retention/deletion filters.

3

Thanks for your time bro!
You're right! I'm gonna do snapshots every month
But first I need to snapshot 3 months logs, I suppose on the config file I need to write

  • filtertype: age
    source: creation_date
    direction: older
    unit: months
    unit_count: 3

Im I right?

And If I want to select an arbitrary month, hows the config file?
Thanks dude

4

This example:

…will only snapshot indices older than 3 months. If you want to snapshot everything from now up through 3 months ago, that's slightly different.

- filtertype: age
  source: creation_date
  direction: older
  unit: days
  unit_count: 1
- filtertype: age
  source: creation_date
  direction: younger
  unit: months
  unit_count: 3

By using 2 filters, the above example takes everything older than a day ago and younger than 3 months ago.

Snapshotting an arbitrary month is done using an absolute period filter:

- filtertype: period
  period_type: absolute
  source: name
  timestring: '%Y.%m.%d'
  unit: months
  date_from: 2019.02
  date_from_format: '%Y.%m'
  date_to: 2019.02
  date_to_format: '%Y.%m'
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.