Docs are stored on same index ¡¡

Elastic Stack Logstash
1

The trouble is that de docs are stored into both index at the same time. Instead to use each own index, one per diferent input.

input {
udp {
host => "192.168.254.60"
port => 5001
type => mikrotik
}
udp {
host => "192.168.254.60"
port => 5000
type => qnap
}
}

filter { ..if [type] == .. }

output {
elasticsearch { hosts => ["192.168.254.50:9200"]
if [type] == "mikrotik" {
index => "mikrotik-%{+YYYY.MM.dd}"
else if [type] == "qnap" {
index => "qnap-%{+YYYY.MM.dd}"
}
}
stdout { codec => rubydebug }
}
}

indexs

I use two diferents inputs - filters and outputs with diferent index too but the documents are stored into both index - the same number of docs. The size its diferent because i use diferent index templates with diferent mappings for especific fields.

2

As you can not have conditionals within a filter the way you do I am surprised anything gets indexed into Elasticsearch.

3

problem solved.
I was using two different configuration files inside the "config.d" folder - one for each input - filter and output with its specific index. I do not understand it well but on having said this I did not end up squared to logstash - that if I gathered data in both ports udp and created different indexes.

If I use a single configuration file for everything - then every log trace is only stored in its correct index.

Thanks for your time :wink: - Christian

input {
udp {
host => "192.168.254.60"
port => 5001
type => mikrotik
}
udp {
host => "192.168.254.60"
port => 5000
type => qnap
}
}

filter {

if [type] == "mikrotik" {

# FIREWALL

if "firewall" in [message] {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => { "message" => "%{MIKROTIKFIREWALL}"}
}
geoip {
source => "src_ip"
target => "geoip"
}
geoip {
source => "dst_ip"
target => "geoip"
}
}

if "2" in [proto] {
mutate {
update => { "proto" => "IGMP" }
}
}

} else if [type] == "qnap" {
grok {
  patterns_dir => ["/etc/logstash/conf.d/patterns"]
  match => { "message" => "%{QNAPSYSLOG}" }
  add_field => [ "received_at", "%{@timestamp}" ]
}
geoip {
source => "src_ip"
target => "geoip"
}
date {
  match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

}

}

output {
if [type] == "mikrotik" {
elasticsearch { hosts => ["192.168.254.50:9200"]
index => "mikrotik-%{+YYYY.MM.dd}"
}
} else if [type] == "qnap" {
elasticsearch { hosts => ["192.168.254.50:9200"]
index => "qnap-%{+YYYY.MM.dd}"
}
}
stdout { codec => rubydebug }
}

4

All files in the config directory are concatenated, so if you do not use conditional to control flow events from all inputs will go to all outputs.