Logstash pipeline output | duplicate messages ending up indexes

Elastic Stack Logstash
4

If the two configurations are completely separate from input to output I would strongly suggest using multiple pipelines. If there is overlap, or you are stuck on an old version then you can use something like

add_field => { inputTopic => "events" }

(with two different value for inputTopic) on the inputs to distinguish them, then use

output {
    if [inputTopic] == "events" {
        elasticsearch {
             ...
        }
    }
}

to send them to different end-points.

Even better, since you are using a kafka input, you can have the input decorate the metadata with the topic name and then make the output configuration conditional upon that.

2 Likes