Two filebeats with almost same data to send to Elastic

Elastic Stack Beats
1

Hello,

I have a setup with 2 syslog servers that get all logs from devices.
Devices send to these 2 servers in parallel, but there is no sync between them. So the syslog servers have "almost" the same data, but I without guarantee (if one is down, the devices continue to log to the second). I cannot modify this "perfectible" setup.

Now I want to send these logs to Elastic, using filebeat and ingest nodes. Here is the setup:

image
image.png476×589 22.1 KB

My issue is that with that setup all logs from my 2 servers will be duplicated in Elastic.
How can I avoid that, having deduplicated logs?

My ideas is setting specific fingerprint. Data will be "only" updated. Is there available without logstash?
Any idea are welcome :wink:

Edit: while writing this, maybe using logstash with file input should be a better alternative, as it's offer more filter/output customization, isn't it?

2
3

Hi,

thanks for the blog post. So I think I must go with logstash to use fingerprint, and it's not available directly on ingest nodes (if I understand well, this refers to https://github.com/elastic/elasticsearch/issues/34085).

Rémi

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.