I am trying to delete old logs using Curator in my Kubernetes Cluster. A sample ES log looks like this:
"_index" : "logstash-2017.09.19",
"_type" : "flb_type",
"_id" : "sample_id",
"_score" : 0.6599215,
"_source" : {
"@timestamp" : "2017-09-19T16:09:04",
"log" : "2017-09-19 16:08:08,521 INFO Preparing Action ID: 1, \"delete_indices\"\n",
"stream" : "stdout",
"time" : "2017-09-19T16:08:08.522064224Z",
"kubernetes" : {
"pod_name" : "curator-1505781600-8b4mv",
"namespace_name" : "default",
"container_name" : "curator",
"docker_id" : "sample_docker_id",
"pod_id" : "sample_pod_id",
"labels" : {
"controller-uid" : "sample_controller_id",
"job-name" : "curator-15057816345"
},
}
}
},
I am trying to run a cronjob that in theory would fire off once a day, and kill any logs older than 2 weeks. For testing I have upped the cron job to run every two minutes, and kill off logs older than 10 minutes. I am trying to use the filtertype: age with source: field_stats to accomplish this, but I am clearly doing something incorrectly because every time the cron job runs it deletes everything. So can someone either help me figure out what bug I have and why the cron job is managing to kill everything, or if there would be a better way to delete everything older than two weeks based on the logs I am collecting.
Here are my configs:
apiVersion: "batch/v2alpha1"
kind: CronJob
metadata:
name: curator
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: curator
image: bobrik/curator
args: ["--config", "/etc/config/config.yml", "/etc/config/action_file.yml"]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: curator-config
restartPolicy: OnFailure
apiVersion: v1
kind: ConfigMap
metadata:
name: curator-config
data:
action_file.yml: |-
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True. If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
1:
action: delete_indices
description: >-
Delete indices older than 5 minutes (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: age
source: field_stats
direction: older
unit: minutes
unit_count: 10
field: '@timestamp'
stats_result: min_value
config.yml: |-
---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
hosts:
- elasticsearch
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']