Hi friends,
I use Es, Logstash and Kibana on a VM and I receive Logs from filebeat on a few VMs.
ES, Logstash and kibana version 7.3.0 and filebeat in 2 version 7.5.2 and 7.6.1 .
I use filebeat Modules like apache and nginx , and multiple pipeline in Logstash.
our filebeats send Logs to logstash. I use also index rollover aliases in elasticsearch output plugin in logstash. my system works really good but the only problem is , new indices created via rollover don't automatically inherit the policy used by the old index. I read this page and I found it is normal. I tried to solve it like description but it doesn't work.
https://www.elastic.co/guide/en/elasticsearch/reference/current/using-policies-rollover.html
I have doubt that I do it right! because I'm not sure about my templates. I used modules to use dynamic Templates in ES. So I didn't define any template by my self. I tried with this Curl command but I know its not for Dynamic templates.
curl -u <user>:<pass> -X PUT "IP:9200/_template/filebeat-7.5.2?pretty" -H 'Content-Type:
application/json' -d'
{
"index_patterns": ["filebeat-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "filebeat",
"index.lifecycle.rollover_alias": "filebeat-7.5.2-nginx-access-default"
}
}
'
and
curl -u <user>:<pass> -X PUT "IP:9200/_template/filebeat-7.6.1?pretty" -H 'Content-Type:
application/json' -d'
{
"index_patterns": ["filebeat-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "filebeat",
"index.lifecycle.rollover_alias": "filebeat-7.6.1-apache-access-default"
}
}
'
we have another aliases for both filebeat-7.5.2 and 7.6.1 and I used similar way for them.
As you can see I used one Policy for all of this aliases. Am I in correct way?
because it doesn't work.
my Logstash filter for example for apache logs :
input {
beats {
port => 5047
client_inactivity_timeout => 10800
}
}
output {
if [event][module] == "apache" {
if [fileset][name] == "access" {
elasticsearch {
ilm_enabled => true
ilm_rollover_alias => "filebeat-7.6.1-apache-access-default"
ilm_pattern => "{now/d}-000001"
ilm_policy => "filebeat"
hosts => "URL:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
bulk_path => "http://URL:9200/_bulk"
pipeline => "%{[@metadata][pipeline]}"
validate_after_inactivity => 300000
user => "user"
password => "Pass"
}
}else if [fileset][name] == "error" {
elasticsearch {
ilm_enabled => true
ilm_rollover_alias => "filebeat-7.6.1-apache-error-pipeline"
ilm_pattern => "{now/d}-000001"
ilm_policy => "filebeat"
hosts => "URL:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
bulk_path => "http://URL:9200/_bulk"
pipeline => "%{[@metadata][pipeline]}"
validate_after_inactivity => 300000
user => "User"
password => "Pass"
}
}
}
}
It's really kind of you if you could help me.