Logs duplicated in all indices

[killbartonking (ire)]

currently running the latest release of the whole stack- at the moment, i have several inputs with a different logstash config file for each, but for some reason, data from one input is showing up in a different index. for example:

/etc/logstash/conf.d/netflow.conf:

input {
udp {
host => "10.5.50.43"
port => 5150
codec => netflow
type => "netflow"
}
}

output {
elasticsearch {
hosts => "10.5.50.42:9200"
manage_template => false
index => "netflow-%{+YYYY.MM.dd}"
}
}

/etc/logstash/conf.d/syslog.conf:

input {
syslog {
host => "10.5.50.43"
port => 5140
}
}

output {
elasticsearch {
hosts => "10.5.50.42:9200"
manage_template => false
index => "syslog-%{+YYYY.MM.dd}"
}
}

if i select the syslog-* index in the discover tab in kibana, i'm able to see netflow data in that index as well. any ideas?

[Evesy (Michael Eves)]

Hey,

Logstash config such as above is essentially all merged together at runtime, so what you have specified as two separate configs will become a single Logstash pipeline. This means any data received form either input (:5140 or :5150) will be sent to both Elasticsearch outputs.

Here's a recent discussion around the same thing: Beats Received on Port 11001 Are Being Processed By Port 11000 Config

Your options are to tag data at the input level, and then wrap your outputs in conditionals that will only match the relevant tags:

e.g.

input {
  beats {
    id => "appa_beats"
    client_inactivity_timeout => 1200
    port => 11000
    tags => ["appa"]
  }
}

output {
  if "appa" in [tags] {
    logstash {
      # The Logstash hosts
      hosts: ["logstash:11000"]
    }
  }
}

You could also look at using Logstash 6.x multiple pipelines: https://www.elastic.co/guide/en/logstash/6.x/multiple-pipelines.html

If you're stuck on an older Logstash version for whatever reason, you can run multiple Logstash instances on the same machine that each load just the correct configs, however I've never tried it.

Cheers,
Mike