I have a setup with 2 syslog servers that get all logs from devices.
Devices send to these 2 servers in parallel, but there is no sync between them. So the syslog servers have "almost" the same data, but I without guarantee (if one is down, the devices continue to log to the second). I cannot modify this "perfectible" setup.
Now I want to send these logs to Elastic, using filebeat and ingest nodes. Here is the setup:
My issue is that with that setup all logs from my 2 servers will be duplicated in Elastic.
How can I avoid that, having deduplicated logs?
My ideas is setting specific fingerprint. Data will be "only" updated. Is there available without logstash?
Any idea are welcome
Edit: while writing this, maybe using logstash with file input should be a better alternative, as it's offer more filter/output customization, isn't it?