Logs duplicated in all indices

Elastic Stack Elasticsearch
2

Hey,

Logstash config such as above is essentially all merged together at runtime, so what you have specified as two separate configs will become a single Logstash pipeline. This means any data received form either input (:5140 or :5150) will be sent to both Elasticsearch outputs.

Here's a recent discussion around the same thing: Beats Received on Port 11001 Are Being Processed By Port 11000 Config

Your options are to tag data at the input level, and then wrap your outputs in conditionals that will only match the relevant tags:

e.g.

input {
  beats {
    id => "appa_beats"
    client_inactivity_timeout => 1200
    port => 11000
    tags => ["appa"]
  }
}

output {
  if "appa" in [tags] {
    logstash {
      # The Logstash hosts
      hosts: ["logstash:11000"]
    }
  }
}

You could also look at using Logstash 6.x multiple pipelines: https://www.elastic.co/guide/en/logstash/6.x/multiple-pipelines.html

If you're stuck on an older Logstash version for whatever reason, you can run multiple Logstash instances on the same machine that each load just the correct configs, however I've never tried it.

Cheers,
Mike