Elasticsearch is multiplying data per *.conf I have

Elastic Stack Elasticsearch
1

Hi,

I'm setting my ELK system and I need to input 3 different *.conf as to input the data from my logs. The problem is that for some reason, when I reboot Logstash, the logs goes to Elasticsearch correctly but it shows data (at Kibana) as many times as indexes with that grok on it.

As I explain myself so bad, I add this screenshot to show you what I see:

image
image1920×973 124 KB

My *.conf configuration are this 4 times just changing directions and ilm* names:

input {
file {
path => "/home/admin/environments/tests/test03/*"
sincedb_path => "/dev/null"
mode => "read"
ignore_older => "10 d"
file_completed_action => "delete"
}
}

filter {
grok {
match => { "message" => "^%{DATA:prex}###############hidden_data##############$"}
}
if ("_grokparsefailure" in [tags]) {
drop{}
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
ilm_rollover_alias => "test3"
ilm_pattern => "000001"
ilm_policy => "test3"
}
stdout { codec => rubydebug }
}

2

When you store multiple configuration files in the config directory these are not treated as separate pipelines but concatenated into a single one. This means that all inputs gets processed by all filters and sent to all outputs unless you use conditionals to control the flow.

3

Now it works perfectly!

Thank you so much.

Hope this issue can be helpful for others :slight_smile:

(New configuration of pipeline now have no "*", instead it contains the hole ".conf" route)