# Logstash sometimes sends duplicate entry to ES **URL:** \ **Category:** Logstash\ **Created:** [July 18, 2017, 3:37pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618 "2017-07-18T15:37:46Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![fozboz](https://avatars.discourse-cdn.com/v4/letter/f/df788c/32.png) [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Post date:** [July 18, 2017, 3:37pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/1 "2017-07-18T15:37:46Z")
Hi there, I have a Metricbeat configured to send Beats to a single Logstash host, which in turn sends to an ES cluster. Metricbeat config: ``` metricbeat.modules: #------------------------------- System Module ------------------------------- - module: system metricsets: - cpu - load - diskio - filesystem - fsstat - memory - network - process enabled: true period: 60s processes: ['.*'] #------------------------------- Apache Module ------------------------------- - module: apache metricsets: ["status"] enabled: true period: 60s hosts: ["http://127.0.0.1"] server_status_path: "server-status" #================================ General ===================================== name: myserver-dev tags: ["test"] #================================ Outputs ===================================== output.logstash: hosts: ["logstash1.private:5044"] loadbalance: true ``` Logstash pipeline: ``` input { beats { port => 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} } geoip { source => "clientip" } } output { elasticsearch { hosts => ["es1.private:9200","es2.private:9200","es3.private:9200"] index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` It all works fine, except that occasionally Logstash will send a duplicate log to ES. The data in the duplicated log is all identical, except for the ID. I think this may have something to do with the ES load balancing. Does anyone have any suggestions how I can diagnose what's going on? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 18, 2017, 5:05pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/2 "2017-07-18T17:05:58Z")
If Beats or Logstash encounter any problems shipping data downstream, they will retry automatically. This means that duplicates can not be avoided in the pipeline. If you however [define an ID based on the content of the event](https://www.elastic.co/blog/logstash-lessons-handling-duplicates) any attempt to write the same event twice would result in an update rather than a duplicate event getting created. ---
**Author:** ![fozboz](https://avatars.discourse-cdn.com/v4/letter/f/df788c/32.png) [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Post date:** [July 18, 2017, 7:13pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/3 "2017-07-18T19:13:38Z")
Thanks for the info! Is there a recommended way of implementing this with Metricbeats? I'm not sure how I would uniquely identify the logs without encapsulating the entire beat somehow. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 18, 2017, 7:52pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/4 "2017-07-18T19:52:17Z")
You can calculate the id in Logstash. ---
**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\ **Post date:** [July 19, 2017, 2:13am UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/5 "2017-07-19T02:13:46Z")
@fozboz This was linked above but here is the explicit link to handle duplicates in Logstash: > **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)** > > Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output. ---
**Author:** ![fozboz](https://avatars.discourse-cdn.com/v4/letter/f/df788c/32.png) [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Post date:** [July 19, 2017, 8:03pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/6 "2017-07-19T20:03:31Z")
I'm going with the "UUID" method to generate the ID in Logstash, which solves the problem shipping from Logstash to ES. Might this still cause a duplicate if Beats encounters a problem shipping to Logstash? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 20, 2017, 5:24am UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/7 "2017-07-20T05:24:03Z")
UUID will only help prevent duplicates being created after it was assigned or if it is taken from an external system. If you use Logstash to create the UUID, you could still end up with duplicates if Beats is forced to retry and this results in duplication of the event. ---
**Author:** ![fozboz](https://avatars.discourse-cdn.com/v4/letter/f/df788c/32.png) [@fozboz](https://discuss.elastic.co/u/fozboz)\ **Post date:** [July 20, 2017, 2:06pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/8 "2017-07-20T14:06:26Z")
So given that the event fields between each Metricbeat are different, how can I achieve this in a more elegant way than e.g. ``` filter { fingerprint { concatenate_sources => true target => "[@metadata][uuid]" method => "MURMUR3" if [@metadata][beat] == "metricbeat" { if [metricset][module] == "system" { if [metricset][name] == "process" { source => ["[every][single]", "[event][field]", "[for][process]", "[listed][individually]", "[in]", "[a][big]", "[long][array]" ] } if [metricset][name] == "diskio" { source => ["[every][single]", "[event][field]", "[for][diskio]", "[listed][individually]", "[in]", "[a][big]", "[long][array]" ] } if [metricset][name] == "network" { source => ["[every][single]", "[event][field]", "[for][network]", "[listed][individually]", "[in]", "[a][big]", "[long][array]" ] } } } } } ``` Is there a way I can say... ``` filter { fingerprint { concatenate_sources => true target => "[@metadata][uuid]" method => "MURMUR3" source => "[the][value][of][every][field][please]" } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 17, 2017, 2:06pm UTC](https://discuss.elastic.co/t/logstash-sometimes-sends-duplicate-entry-to-es/93618/9 "2017-08-17T14:06:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.