I can use ilm to rollover index by date on none prod env, like this:
POST rehearsal-k8s-logs/_rollover?dry_run
{
"conditions" : {
"max_age": "1d",
"max_size": "20gb"
}
}
---
{
"acknowledged" : false,
"shards_acknowledged" : false,
"old_index" : "rehearsal-k8s-logs-rehearsal-kafka-rehearsal-application-kube-2020.06.07-000026",
"new_index" : "rehearsal-k8s-logs-rehearsal-kafka-rehearsal-application-kube-2020.06.08-000027",
"rolled_over" : false,
"dry_run" : true,
"conditions" : {
"[max_age: 1d]" : false,
"[max_size: 20gb]" : false
}
}
but when I copy the config file to prod env, the date can't be rollover on the second day, like this:
POST prod-k8s-logs/_rollover?dry_run
{
"conditions" : {
"max_age": "1d",
"max_size": "20gb"
}
}
---
{
"acknowledged" : false,
"shards_acknowledged" : false,
"old_index" : "prod-k8s-logs-kafka-prod-app-kube-2020.06.04-000054",
"new_index" : "prod-k8s-logs-kafka-prod-app-kube-2020.06.04-000055",
"rolled_over" : false,
"dry_run" : true,
"conditions" : {
"[max_age: 1d]" : false,
"[max_size: 20gb]" : false
}
}
I want to rollover a new index with new name like this:
"prod-k8s-logs-kafka-prod-app-kube-2020.06.05-000054"
but I get a new index name with this:
"prod-k8s-logs-kafka-prod-app-kube-2020.06.04-000054"
my config:
filebeat.yaml
filebeat.inputs:
- type: kafka
hosts:
- wn0.kafka.com:9092
- wn1.kafka.com:9092
- wn2.kafka.com:9092
topics: ["k8s-pod-logs-prod-kafka-topic"]
group_id: "fb-k8s-pod-prod-group"
processors:
- rename:
fields:
- from: "message"
to: "tempmessage"
- decode_json_fields:
fields: "tempmessage"
target: ""
max_depth: 2
- drop_fields:
fields: ["tempmessage","kafka"]
ignore_missing: true
output.elasticsearch:
hosts: ["172.21.25.80:9200"]
username: "elastic"
password: "123456"
bulk_max_size: 1024
xpack.monitoring:
enabled: true
setup.ilm.enabled: true
setup.ilm.rollover_alias: prod-k8s-logs
setup.ilm.pattern: kafka-prod-app-kube-{now/d}-000001
setup.ilm.policy_name: prod-k8s-log-ilm-policy
setup.ilm.overwrite: true
setup.ilm.policy_file: "/usr/share/filebeat/ilm-policy.json"
sh-4.2# cat /usr/share/filebeat/ilm-policy.json
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1d",
"max_size": "20G"
}
}
},
"warm": {
"min_age": "1d",
"actions": {
"allocate": {
"number_of_replicas": 0,
"include": {},
"exclude": {}
},
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "30d",
"actions": {
"allocate": {
"number_of_replicas": 0,
"include": {},
"exclude": {}
},
"freeze": {},
"set_priority": {
"priority": 0
}
}
},
"delete": {
"min_age": "60d",
"actions": {
"delete": {}
}
}
}
}
}
ES version:7.4.2
filebeat version: 7.4.2
