# Two filebeats with almost same data to send to Elastic

**URL:** <https://discuss.elastic.co/t/two-filebeats-with-almost-same-data-to-send-to-elastic/201079>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 25, 2019, 4:00pm UTC](https://discuss.elastic.co/t/two-filebeats-with-almost-same-data-to-send-to-elastic/201079 "2019-09-25T16:00:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rverchere](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@rverchere](https://discuss.elastic.co/u/rverchere)\
**Post date:** [September 25, 2019, 4:00pm UTC](https://discuss.elastic.co/t/two-filebeats-with-almost-same-data-to-send-to-elastic/201079/1 "2019-09-25T16:00:18Z")

</div>

Hello,

I have a setup with 2 syslog servers that get all logs from devices.  
Devices send to these 2 servers in parallel, but there is no sync between them. So the syslog servers have "almost" the same data, but I without guarantee (if one is down, the devices continue to log to the second). I cannot modify this "perfectible" setup.

Now I want to send these logs to Elastic, using filebeat and ingest nodes. Here is the setup:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/5/85e0a3158c34478c1db25144e1f18cdf4eafdac8.png)

My issue is that with that setup all logs from my 2 servers will be duplicated in Elastic.  
How can I avoid that, having deduplicated logs?

My ideas is setting specific fingerprint. Data will be "only" updated. Is there available without logstash?  
Any idea are welcome 😉

Edit: while writing this, maybe using logstash with file input should be a better alternative, as it's offer more filter/output customization, isn't it?

---

<div class="post-metadata">

**Author:** ![kumarabhi](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kumarabhi](https://discuss.elastic.co/u/kumarabhi)\
**Post date:** [September 26, 2019, 5:48pm UTC](https://discuss.elastic.co/t/two-filebeats-with-almost-same-data-to-send-to-elastic/201079/2 "2019-09-26T17:48:03Z")

</div>

> **[Efficient Duplicate Prevention for Event-Based Data in Elasticsearch](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch)**
>
> Need to prevent duplicates in the Elastic Stack with minimal performance impact? In this blog we look at different options in Elasticsearch and provide some practical guidelines.

---

<div class="post-metadata">

**Author:** ![rverchere](https://avatars.discourse-cdn.com/v4/letter/r/dbc845/32.png) [@rverchere](https://discuss.elastic.co/u/rverchere)\
**Post date:** [October 2, 2019, 1:57pm UTC](https://discuss.elastic.co/t/two-filebeats-with-almost-same-data-to-send-to-elastic/201079/3 "2019-10-02T13:57:08Z")

</div>

Hi,

thanks for the blog post. So I think I must go with logstash to use fingerprint, and it's not available directly on ingest nodes (if I understand well, this refers to [https://github.com/elastic/elasticsearch/issues/34085](https://github.com/elastic/elasticsearch/issues/34085)).

Rémi

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 30, 2019, 1:57pm UTC](https://discuss.elastic.co/t/two-filebeats-with-almost-same-data-to-send-to-elastic/201079/4 "2019-10-30T13:57:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.