Beat ILM Problems

Dear All.

I have following configuration.
Elasticsearch Version 7.4.0

We have several beats in use.
filebeat, metricbeat, packetbeat, auditbeat and heartbeat

We have following to communication streams from the beats to elasticsearch.

beat -> elasticsearch

beat -> logstash (with input beat port 5044 and output elasticsearch) -> elasticsearch

logstash beat configuration:

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => ["${ES_HOSTS}"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

The beats that are communicating direct to elasticsearch are writing their data into the

filebeat-7.4.0-YEAR.MONTH.DAY-000001
metricbeat-7.4.0-YEAR.MONTH.DAY-000001
packetbeat-7.4.0-YEAR.MONTH.DAY-000001
auditbeat-7.4.0-YEAR.MONTH.DAY-000001
heartbeat-7.4.0-YEAR.MONTH.DAY-000001

The beats that are communicating to logstash creating every day a new index

filebeat-7.4.0-YEAR.MONTH.DAY
metricbeat-7.4.0-YEAR.MONTH.DAY
packetbeat-7.4.0-YEAR.MONTH.DAY
auditbeat-7.4.0-YEAR.MONTH.DAY
heartbeat-7.4.0-YEAR.MONTH.DAY

I have following problems with the ILM

Index lifecycle error:
illegal_argument_exception: index.lifecycle.rollover_alias [filebeat-7.4.0] does not point to index [filebeat-7.4.0-2020.01.28]
illegal_argument_exception: index.lifecycle.rollover_alias [metricbeat-7.4.0] does not point to index [metricbeat-7.4.0-2020.01.27]
illegal_argument_exception: index.lifecycle.rollover_alias [packetbeat-7.4.0] does not point to index [packetbeat-7.4.0-2020.02.09]
illegal_argument_exception: index.lifecycle.rollover_alias [auditbeat-7.4.0] does not point to index [auditbeat-7.4.0-2020.01.31]
illegal_argument_exception: index.lifecycle.rollover_alias [heartbeat-7.4.0] does not point to index [heartbeat-7.4.0-2020.01.27]

I think this means that the beats that are communicationg over logstash are not able to use the Index Lifecycle Policies, because the indexes have not the extension -000001.

I will like to see that also the beats that are communicating over logstash are writing there data into the indexes with the extension -000001.

How can I fix this problem?

best regards
Alexander

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.