Syslogs, Beat, Redundancy and Architecture

[jasonh (Jason)]

Hello Group
I'm setting up an ELK which will take a feed of syslogs from a variety of devices including Cisco's Linux Server Syslogs, App files and Windows server logs and files.
I want to make it as redundant as possible so that we can send syslogs from the network devices (and servers etc) to two aggregation points but avoid duplication of lines.

So for syslogs I am defining the devices to send to two syslog-ng nodes. The part I need to work out is how I can then get both syslog-ng nodes to send to logstash (2 nodes) but not duplicate the log entries.
Is there a mechanism I can use so I can have a beat(?) on each syslog-ng sending to logstash (using the redundancy option) but avoid getting duplicated syslogs in the ES at the final aggregation.

[andrewkroh]

One way would be to fingerprint the event at each Logstash aggregation point (filebeat -> logstash -> elasticsearch). Then use the fingerprint as the document_id in the Logstash elasticsearch output. This way a given message will be assigned the same ID at both aggregations points and Elasticsearch will only index the first one that is inserted.