# Curator - need help to delete old logs with timestamp

**URL:** <https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058>\
**Category:** Elasticsearch\
**Created:** [September 19, 2017, 5:08pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058 "2017-09-19T17:08:56Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![DStorck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dstorck/32/33811_2.png) [@DStorck](https://discuss.elastic.co/u/DStorck)\
**Post date:** [September 19, 2017, 5:08pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/1 "2017-09-19T17:08:56Z")

</div>

I am trying to delete old logs using Curator in my Kubernetes Cluster. A sample ES log looks like this:

```auto
        "_index" : "logstash-2017.09.19",
        "_type" : "flb_type",
        "_id" : "sample_id",
        "_score" : 0.6599215,
        "_source" : {
          "@timestamp" : "2017-09-19T16:09:04",
          "log" : "2017-09-19 16:08:08,521 INFO Preparing Action ID: 1, \"delete_indices\"\n",
          "stream" : "stdout",
          "time" : "2017-09-19T16:08:08.522064224Z",
          "kubernetes" : {
            "pod_name" : "curator-1505781600-8b4mv",
            "namespace_name" : "default",
            "container_name" : "curator",
            "docker_id" : "sample_docker_id",
            "pod_id" : "sample_pod_id",
            "labels" : {
              "controller-uid" : "sample_controller_id",
              "job-name" : "curator-15057816345"
            },
          }
        }
      },

```

I am trying to run a cronjob that in theory would fire off once a day, and kill any logs older than 2 weeks. For testing I have upped the cron job to run every two minutes, and kill off logs older than 10 minutes. I am trying to use the `filtertype: age` with `source: field_stats` to accomplish this, but I am clearly doing something incorrectly because every time the cron job runs it deletes everything. So can someone either help me figure out what bug I have and why the cron job is managing to kill everything, or if there would be a better way to delete everything older than two weeks based on the logs I am collecting.  
Here are my configs:

```auto
apiVersion: "batch/v2alpha1"
kind: CronJob
metadata:
  name: curator
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: curator
            image: bobrik/curator
            args: ["--config", "/etc/config/config.yml", "/etc/config/action_file.yml"]
            volumeMounts:
              - name: config-volume
                mountPath: /etc/config
          volumes:
            - name: config-volume
              configMap:
                name: curator-config
          restartPolicy: OnFailure

```

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: curator-config
data:
  action_file.yml: |-
    ---
    # Remember, leave a key empty if there is no value. None will be a string,
    # not a Python "NoneType"
    #
    # Also remember that all examples have 'disable_action' set to True. If you
    # want to use this action as a template, be sure to set this to False after
    # copying it.
    actions:
      1:
        action: delete_indices
        description: >-
          Delete indices older than 5 minutes (based on index name), for logstash-
          prefixed indices. Ignore the error if the filter does not result in an
          actionable list of indices (ignore_empty_list) and exit cleanly.
        options:
          timeout_override:
          continue_if_exception: False
          disable_action: False
        filters:
        - filtertype: age
          source: field_stats
          direction: older
          unit: minutes
          unit_count: 10
          field: '@timestamp'
          stats_result: min_value

  config.yml: |-
    ---
    # Remember, leave a key empty if there is no value. None will be a string,
    # not a Python "NoneType"
    client:
      hosts:
        - elasticsearch
      port: 9200
      url_prefix:
      use_ssl: False
      certificate:
      client_cert:
      client_key:
      ssl_no_validate: False
      http_auth:
      timeout: 30
      master_only: False

    logging:
      loglevel: INFO
      logfile:
      logformat: default
      blacklist: ['elasticsearch', 'urllib3']

```

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 19, 2017, 5:25pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/2 "2017-09-19T17:25:41Z")

</div>

> [@DStorck](#):
>
> ```auto
> - filtertype: age
> source: field_stats
> direction: older
> unit: minutes
> unit_count: 10
> field: '@timestamp'
> stats_result: min_value
> 
> ```

Using `field_stats` in this way means that if the _smallest_ value of `@timestamp` in the index is more than 10 minutes old, it will delete the entire index. It doesn't sound like that is what you want.

Also, as this is the _only_ filter, it will delete _all_ indices matching this criteria. You probably don't want to delete your `.kibana` index, or perhaps other, similar indices. It would be best to limit an `age` filter to indices matching a given pattern first, and then apply the `age` filter.

---

<div class="post-metadata">

**Author:** ![DStorck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dstorck/32/33811_2.png) [@DStorck](https://discuss.elastic.co/u/DStorck)\
**Post date:** [September 19, 2017, 5:30pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/3 "2017-09-19T17:30:06Z")

</div>

Really the goal was to delete all logs older than two weeks. I had looked into the age filter and tried this one I found:

```auto
        filters:
        - filtertype: pattern
          kind: prefix
          value: logstash-
          exclude: False
        - filtertype: age
          source: name
          direction: older
          timestring: '%Y.%m.%d'
          unit: minutes
          unit_count: 10
          exclude: False

```

but was running into the same problem. Based on what I have available in my logs what would you suggest is the best way to achieve this goal?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 19, 2017, 6:36pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/4 "2017-09-19T18:36:53Z")

</div>

> [@DStorck](#):
>
> ```auto
> filters:
> - filtertype: pattern
> kind: prefix
> value: logstash-
> exclude: False
> - filtertype: age
> source: name
> direction: older
> timestring: '%Y.%m.%d'
> unit: minutes
> unit_count: 10
> exclude: False
> 
> ```

The same thing applies here, though the method of age calculation is derived from a timestamp in the index name instead. Any index older than 10 minutes ago would be deleted.

```auto
        filters:
        - filtertype: pattern
          kind: prefix
          value: logstash-
          exclude: False
        - filtertype: age
          source: field_stats
          direction: older
          unit: days
          unit_count: 14
          field: '@timestamp'
          stats_result: min_value

```

Something like this is more likely to be what you want. This example will delete `logstash-` prefixed indices older than 14 days.

---

<div class="post-metadata">

**Author:** ![DStorck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dstorck/32/33811_2.png) [@DStorck](https://discuss.elastic.co/u/DStorck)\
**Post date:** [September 20, 2017, 11:23pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/5 "2017-09-20T23:23:03Z")

</div>

This seems like it's working, thanks so much.

---

<div class="post-metadata">

**Author:** ![DStorck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dstorck/32/33811_2.png) [@DStorck](https://discuss.elastic.co/u/DStorck)\
**Post date:** [September 21, 2017, 5:51pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/6 "2017-09-21T17:51:43Z")

</div>

Follow up (and i'm happy to create a new topic , but this seems related ) :  
I got the filter working, and yesterday when I ran it, it went ahead and deleted all logs from index logstash-2017.09.19. ( I ran this on the 20th, so it deleted all older logs perfectly) . However, it didn't seem to actually delete the index, as I noted this morning when I ran /\_cat/indices and saw:

```auto
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2017.09.20 vQ06hfLuSRmrctDt-Ejkyg 5 1 2264044 0 3.4gb 1.7gb
green open logstash-2017.09.21 lHspQ9QoR7eAaX4H_IhTdg 5 1 3142314 0 5.9gb 2.9gb
green open logstash-2017.09.19 kuRXxBRLSJywe1Qi9DsguA 5 1 0 0 1.2kb 650b

```

This caused a problem for curator, as it is now failing with : `Failed to complete action: delete_indices. <class 'curator.exceptions.ActionError'>: Field "@timestamp" not found in index "logstash-2017.09.19"` , and because of that error, nothing else will run. I have already tried including `ignore_empty_list: True` in my config options, but I'm still getting the same error.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 21, 2017, 6:06pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/7 "2017-09-21T18:06:25Z")

</div>

Curator cannot delete only documents and leave an index behind because it only makes use of the delete index API call (in this instance). Something else seems to have happened here. Without more information, I cannot explain what transpired. I can tell you, though, that some other process created that index, or something went wrong in your cluster and it attempted to recover from whatever that was, and tried to recreate the index, but only managed to make the structure (i.e., index, but no documents in it).

Do you make use of a process that creates indices on a time schedule?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2017, 6:06pm UTC](https://discuss.elastic.co/t/curator-need-help-to-delete-old-logs-with-timestamp/101058/8 "2017-10-19T18:06:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.