# Logstash x handling duplicate

**URL:** <https://discuss.elastic.co/t/logstash-x-handling-duplicate/319604>\
**Category:** Logstash\
**Created:** [November 23, 2022, 2:19am UTC](https://discuss.elastic.co/t/logstash-x-handling-duplicate/319604 "2022-11-23T02:19:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lchan/32/90857_2.png) [@lchan](https://discuss.elastic.co/u/lchan)\
**Post date:** [November 23, 2022, 2:19am UTC](https://discuss.elastic.co/t/logstash-x-handling-duplicate/319604/1 "2022-11-23T02:19:40Z")

</div>

Hi all,

I am trying to increase number of logstash servers for redundancy and want to know if using fingerprint would achieve it.

Does this basically sending the same stream of logs/messages via multiple logstash servers and ES will only update existing document contents after comparing the fingerprint?

Also, a rather dumb question, the document\_id are saving in memory? Just want to under the implication on the performance as I didn't find much documentation on the internal.

> **[Efficient Duplicate Prevention for Event-Based Data in Elasticsearch](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch)**
>
> Need to prevent duplicates in the Elastic Stack with minimal performance impact? In this blog we look at different options in Elasticsearch and provide some practical guidelines.

```auto
filter {
  fingerprint {
    source => "message"
    target => "[@metadata][fingerprint]"
    method => "SHA1",
    key => "Log analytics",
    base64encode => true
  }
}
output {
  elasticsearch {
    hosts => "myes.com"
    document_id => "%{[@metadata][fingerprint]}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 23, 2022, 4:01am UTC](https://discuss.elastic.co/t/logstash-x-handling-duplicate/319604/2 "2022-11-23T04:01:35Z")

</div>

> [@lchan](#):
>
> I am trying to increase number of logstash servers for redundancy and want to know if using fingerprint would achieve it.
> 
> Does this basically sending the same stream of logs/messages via multiple logstash servers and ES will only update existing document contents after comparing the fingerprint?

If you send the same messages through three logstash servers, all writing to the same index with the same document\_id, then you more than triple the cost. Three logstash servers triples the cost, and re-indexing or searching to confirm the incoming document does not exist increases the cost on the elasticsearch side.

Personally I would put a kafka cluster in front of logstash, in the hope that consumers would have been rebalanced. Other pub/sub products could probably balance the load across logstash consumers equally well.

This is one of those trade-offs that everyone in engineering makes all the time. You didn't describe any of the reliability constraints or the cost constraints. These (and other) constraints are what we balance every day.

You need a message consumer that acks the message that it consumes and provides data integrity (logstash can do it with PQs, DLQs for some pipelines). As well as a message producer that respects acks.

You have not provided enough info for us to provide support..

---

<div class="post-metadata">

**Author:** ![lchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lchan/32/90857_2.png) [@lchan](https://discuss.elastic.co/u/lchan)\
**Post date:** [November 23, 2022, 5:27pm UTC](https://discuss.elastic.co/t/logstash-x-handling-duplicate/319604/3 "2022-11-23T17:27:26Z")

</div>

Hi @badger,

> If you send the same messages through three logstash servers, all writing to the same index with the same document\_id, then you more than triple the cost. Three logstash servers triples the cost, and re-indexing or searching to confirm the incoming document does not exist increases the cost on the elasticsearch side.

Thanks for the suggestion and insight. I am aware adding logstash servers would add compute cost. (We thought about implementing Kafka but that's another consideration as our syslog server spawn in 8 different datacenters.) The short terms is to provide redundancy on our syslog pipleline.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/6/4612232708523c0028d4ca8ed49bb377efade33b.png)

Do we actually want PQ in this case if there multiple logstash servers pushing the messages? For example - logstashA rebooted; logstashB would already wrote the the same index.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2022, 5:27pm UTC](https://discuss.elastic.co/t/logstash-x-handling-duplicate/319604/4 "2022-12-21T17:27:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.