Logstash with input from file and input from beats (port 5044) create two indices with the same data

Good morning, I have two configuration files in Logstash conf directory:

First file:

File: firewall.conf
** Logs from Firewall **

input {
file {
path => "/var/log/remote/firewall/*-firewall.log"
ignore_older => 60
tags => ["firewall-log"]
}
}
...
...
...
output {
if "firewall-log" in [tags] {
elasticsearch {
index => "firewall-log-%{+YYYY.MM}"
hosts => ["localhost: 9200"]
}
}

Second file:

** File: beats.conf **
** Logs from auditlogbeat and winlogbeat **

input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
document_type => "%{[@metadata][type]}"
}
}

With this configuration, two indexes for the firewall data appear in Kibana. The first of them firewall-log-2019-07 with the data of the log file specified in the input section of firewall.conf. The funny thing is that a second index called %{[@metadata] [beat]}-{[@metadata] [version]}-%{2019-07} is created and with the same data as firewall-log-2019.07 index. I don't undestand the reason because in the input section of beats.conf I specified beats and listen from port 5044 and however Logstash is taking data from the firewall log file.
To solve the problem I have changed the output section in beats.conf file and I have added a if clause:

output {
if "beats_input_codec_plain_applied" in [tags] {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
document_type => "%{[@metadata][type]}"
}
}
}

I don't know If this is a good solution, but this way the index %{[@metadata] [beat]}-{[@metadata] [version]}-% {2019-07} is not created and the data from the firewall log file is loaded only in the firewall-log-2019.07 index.

Thank you!!

Please read this post.

Now I got it. Thank you for your help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.