Two filebeats with almost same data to send to Elastic


I have a setup with 2 syslog servers that get all logs from devices.
Devices send to these 2 servers in parallel, but there is no sync between them. So the syslog servers have "almost" the same data, but I without guarantee (if one is down, the devices continue to log to the second). I cannot modify this "perfectible" setup.

Now I want to send these logs to Elastic, using filebeat and ingest nodes. Here is the setup:

My issue is that with that setup all logs from my 2 servers will be duplicated in Elastic.
How can I avoid that, having deduplicated logs?

My ideas is setting specific fingerprint. Data will be "only" updated. Is there available without logstash?
Any idea are welcome :wink:

Edit: while writing this, maybe using logstash with file input should be a better alternative, as it's offer more filter/output customization, isn't it?


thanks for the blog post. So I think I must go with logstash to use fingerprint, and it's not available directly on ingest nodes (if I understand well, this refers to


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.