I have read almost all the topics on the Elastic page (tutorials) and topics from this forum, but I still have doubts how to set these settings.
The policy on the life cycle of the index and Rolling over.
Logstash config:
input {
beats {
port => 5044
}
filter {
if "tracking" in [tags] {
grok {
patterns_dir => ["/opt/logstash/patterns"]
match => {"message" => "%{MEMORY}"}
add_tag => ["memory"]
}
grok {
patterns_dir => ["/opt/logstash/patterns"]
match => {"message" => "%{CPU}"}
add_tag => [ "cpu"]
}
grok {
patterns_dir => ["/opt/logstash/patterns"]
match => {"message" => "%{NUMBER:app_time}%{COMA}%{APP:app_type}%{COMA}%{MSG:app_msg}%{END}"}
}
} else if "serial_log" in [tags] {
grok {
match => { "source" => "%{IP:[host][name]}"}
}
}
mutate {
remove_tag => ["_grokparsefailure", "beats_input_codec_plain_applied"]
}
}
}
output {
elasticsearch {
hosts => ["localhost"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
each index in above config is created each day with new date i.e : filebeat-2019.04.24 , filebeat-2019.04.25 and so on. But with time I have these indexes over 30 (e.g. after a month) and I would like them to be automatically removed e.g. after 14 days or size more than e.g. 20gb. That's why I have tried to use Index lifecycle policy. But still I cannot figure it out how this feature works
I would like to create based on my config output to ILM based on example from elastic site:
output {
elasticsearch {
ilm_rollover_alias: "filbeat"
ilm_pattern:
ilm_policy: "test_policy"
}
}
- Should I set: ilm_enabled => true
- Should I set above parameters i.e: ilm_rollover_alias: "custom" or rather ilm_rollover_alias => "custom"
- Sould I kept my index name as it is now: index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" , which give index named i.e: filebeat-2019.04.25 or point index to something else ?
- Should I set any index alias ?
- What I should set in ilm_pattern
- Should I use any index template ?
From Elastic/Kibana point of view i did:
Create policy:
{
"test_policy" : {
"version" : 1,
"modified_date" : "2019-04-25T11:59:26.843Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "20gb",
"max_age" : "14d"
}
}
},
"delete" : {
"min_age" : "2d",
"actions" : {
"delete" : { }
}
}
}
}
}
}
Create Template:
{
"test_policy" : {
"order" : 10,
"index_patterns" : [
"filebeat-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "test_policy"
},
"routing" : {
"allocation" : {
"require" : {
"data" : "hot"
}
}
}
}
},
"mappings" : { },
"aliases" : { }
}
}
- Should I add any other options or point to something ?
I did a tutorial from: tutorial_elastic_index_lifecycle
but with no effect.