# Duplicate events with filebeat -\> logstash -\> elasticsearch pipeline **URL:** \ **Category:** Logstash\ **Created:** [October 26, 2017, 7:01am UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348 "2017-10-26T07:01:55Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 26, 2017, 7:01am UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/1 "2017-10-26T07:01:55Z")
I have below pipeline to parse the log files written by an application and send them to elasticsearch : filebeat -\> logstash -\> elasticsearch For some time my elasticsearch node went down. When I restarted the elasticsearch after around 30 mins, I observed that there are multiple entries for some log messages in Kibana. This happened for 30 log messages in my case. What may be the reason for this? filebeat config : filebeat.prospectors: - input\_type: log output.logstash: hosts: ["localhost:5044"] Logstash Config : input { beats { port =\> 5044 } } filter { ruby { code=\> "event.set('read\_timestamp', Time.new)" } mutate { remove\_field =\> ["beat", "tags", "input\_type"] } } output { elasticsearch { hosts =\> "localhost:9200" index =\> "filebeat-logstash-%{+YYYY.MM.dd}" } } ---
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 30, 2017, 7:40am UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/2 "2017-10-30T07:40:53Z")
Any pointers here would be very helpful. I have captured the debug logs from logstash. Please let me know how do I post the file/logs for further analysis. The size of the logs is huge don't want to paste here. ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [October 30, 2017, 2:35pm UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/3 "2017-10-30T14:35:51Z")
The way ElasticSearch decides document uniqueness is it's \__id_. By default, if none is provided, ElasticSearch will auto-generate one itself upon receiving any document insert request. So if you just send the same document multiple times it will be indexed as separate documents. So, this is probably because you do not set a unique \__id_ yourself. If you want to avoid it you should create a unique id for each document and [supply](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id) it to the elasticsearch output. As far as I know ID generation is not yet supported in beats components, as per [https://github.com/elastic/beats/issues/5269](https://github.com/elastic/beats/issues/5269), but you can probably use Logstash's [UUID filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html) or just construct your own. ---
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 30, 2017, 4:25pm UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/4 "2017-10-30T16:25:50Z")
@paz Thanks for the suggestion. I will try that out. I would like to understand how the pipeline works. Why logstash or filebeat (via logstash) is sending the event multiple times to elasticsearch. ---
**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\ **Post date:** [October 31, 2017, 2:06pm UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/5 "2017-10-31T14:06:56Z")
I assume it has to do something with Logstash trying to be fault-tolerant. It might be possible that when ElasticSearch went down, it indexed the last bulk request (or a part of it), but Logtash never received acknowledgement of a successful operation, so it kept trying to reprocess that bulk request. When eventually ES came back up, it received the request again but treating it as a new batch of documents (since there was no unique identifier) causing some documents to be indexed twice. ---
**Author:** ![an.secure](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@an.secure](https://discuss.elastic.co/u/an.secure)\ **Post date:** [October 31, 2017, 4:42pm UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/6 "2017-10-31T16:42:27Z")
In my case I am seeing the events multiple times not just twice. some events even got to elasticsearch 6 times. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 28, 2017, 4:42pm UTC](https://discuss.elastic.co/t/duplicate-events-with-filebeat-logstash-elasticsearch-pipeline/105348/7 "2017-11-28T16:42:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.