Noob Help with duplicates

not sure about the [message][type} if statements, as the message field is using a string, and I don't see you converting it with the [json] filter. Though I am not familiar with your data format.

I think there might be ways of simplifying your output section to be easier to read
For examole, on every Inputut I add a field called "dst_index" then in my elasticsearch output

I can do

if [dst_indes] =~ /heartbeat/

elkasticsearch{}

else

elasticsearch{
index => "%{dst_index}"

}

This allows me to treat the one heartbeat index differently

Also I like to add a tag to my input / filters and heck even outputs when debugging so I can see what block is processing the data.

hope this gives you some idea's

oh and if data is duplicated it is probably hitting multiple output statements