I please need help to understand ILM in combination with Logstash. My current setup somehow seems not to work as expected. Using ELK 7.9.1 - My configuration looks like:
ILM policy - logstash-syslog-compliance
{ -
"logstash-syslog-compliance": { -
"version": 2,
"modified_date": "2020-10-28T06:47:07.463Z",
"policy": { -
"phases": { -
"hot": { -
"min_age": "0ms",
"actions": { -
"rollover": { -
"max_size": "50gb",
"max_age": "1d"
},
"set_priority": { -
"priority": 100
}
}
},
"delete": { -
"min_age": "30d",
"actions": { -
"delete": { -
"delete_searchable_snapshot": true
}
}
}
}
}
}
}
Before I also had a warm and cold phase, but as I later read this seems not to work if you don't have explicit warm and cold nodes, so I removed the phases and now only have hot and delete. Still yesterday's index didn't get rolled over automatically - described below in detail.
Index Template
{ -
"index_templates": [ -
{ -
"name": "logstash-syslog-compliance",
"index_template": { -
"index_patterns": [ -
"compliance_logstash_syslog*"
],
"composed_of": [ -
"syslog-settings-compliance",
"syslog-mappings-dynamic",
"syslog-mappings"
],
"version": 1
}
}
]
}
Component Template - syslog-settings-compliance
{ -
"component_templates": [ -
{ -
"name": "syslog-settings-compliance",
"component_template": { -
"template": { -
"settings": { -
"index": { -
"lifecycle": { -
"name": "logstash-syslog-compliance",
"parse_origination_date": "true",
"rollover_alias": "compliance_logstash_syslog"
},
"routing": { -
"allocation": { -
"total_shards_per_node": "1"
}
},
"mapping": { -
"total_fields": { -
"limit": "10000"
}
},
"refresh_interval": "15s",
"number_of_shards": "1",
"translog": { -
"durability": "async"
},
"auto_expand_replicas": "false",
"max_docvalue_fields_search": "200",
"number_of_replicas": "1"
}
}
},
"version": 1
}
}
]
}
Logstash's elasticsearch output - configured on 2 filter-nodes
elasticsearch {
id => "elasticsearch_compliance_logstash_syslog"
# as we have our own template configured we don't want to let logstash manage the template
manage_template => false
codec => json_lines
cacert => "/etc/logstash/certs/elasticsearch-ca.pem"
# user has cluster permissions: monitor, manage_ilm
# user has indices permissions: create, write, manage_ilm, delete, create_index, manage
user => "logstash_internal"
password => "supersecureandsecret"
ssl => true
ilm_enabled => "true"
ilm_rollover_alias => "compliance_logstash_syslog"
ilm_pattern => "{now/d}-000001"
ilm_policy => "logstash-syslog-compliance"
# obfuscated IPs
hosts => [ "https://123.123.123.1:9200", "https://123.123.123.2:9200" ]
}
Problem
So today I saw that the rollover again did not work. I still had an active index compliance_logstash_syslog-2020.10.27-000001. After reading that hot, warm, cold and delete seems not to work without explicitly set nodes (it's only a small test-cluster), I changed the policy to hot and delete only. Still the index didn't get rolled over after waiting half an hour plus (the poll interval is default at 10min). The compliance_logstash_syslog-2020.10.27-000001/_ilm/explain?human showed it's age is 1.27d. The size was definitely not hit - but the age. The "explain" didn't show any errors.
So I tried a manual rollover compliance_logstash_syslog/_rollover/compliance_logstash_syslog-2020.10.28-000001 with the same conditions and the rollover worked without any problem. But now I wonder if it will do the rollover tomorrow to compliance_logstash_syslog-2020.10.29-000001. What's wrong with my setup, what am I missing?
Also I wonder - why is the alias not removed from old indices, imho this is error-prone even with the write-index parameter. It's strange that you have to create the first index with the write-index alias manually. Can I configure that somewhere within a ILM policy to remove aliases from old indices?
Currently I'm not seeing real advantages of ILM compared to curator configs and scripts.
