Beat not creating index the right way after change output from elasticsearch to logstash

Elastic Stack Logstash
1

hi

installed new elastic stack with version 7.9.1 and filebeat localy
everything working fine with default config

installed logstash and changed the output in filebeat.yml to logstash index creating not working as before in default config.

filebeat.yml:

# ------------------------------ Logstash Output -------------------------------
 output.logstash:
   # The Logstash hosts
   hosts: ["localhost:5044"]
/etc/logstash/conf.d/30-elasticsearch-output.conf:

 output {
   if [@metadata][pipeline] {
     elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
     pipeline => "%{[@metadata][pipeline]}"
     }
   } else {
     elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
     }
   }
 }

then I see that the created index not having -000001 at the end and i changed the config (not sure that's the right way) to:

/etc/logstash/conf.d/30-elasticsearch-output.conf:

output {
      if [@metadata][pipeline] {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}-000001"
        pipeline => "%{[@metadata][pipeline]}"
        }
      } else {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}-000001"
        }
      }
    }

now the index name is created correctly but if I check the index, the Aliases is set to none:

2020-09-25 11_31_49-Window
2020-09-25 11_31_49-Window1204×496 59.4 KB

and after some minutes it display the following error:

illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.9.1] does not point to index [filebeat-7.9.1-2020.09.25-000001]

so i decided to change the index-template "filebeat-7.9.1" and add the following config in Aliases:

    {
      "filebeat-7.9.1": {}
    }

then after that config, restarted filebeat and logstash services and deleted all indexes with name filebeat* manually and the Aliases is set to "filebeat-7.9.1" but after some minutes I received the following index lifecycle error:

illegal_argument_exception: Rollover alias [filebeat-7.9.1] can point to multiple indices, found duplicated alias [[filebeat-7.9.1]] in index template [filebeat-7.9.1]

so I decided to delete the Aliases config again in index template...

what I'm doing wrong? if output directly to elasticsearch it takes the right ilm-config, set the right alias and everything is ok. why that is not working through logstash and it takes the same config as point output directly to elasticsearch?

Thank you very much for helping me out of confused about ILM, templates and aliases

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.