# Filebeat lit des lignes 2 fois

**URL:** https://discuss.elastic.co/t/filebeat-lit-des-lignes-2-fois/236452
**Category:** Discussions en français
**Created:** [June 10, 2020, 7:31am UTC](https://discuss.elastic.co/t/filebeat-lit-des-lignes-2-fois/236452 "2020-06-10T07:31:12Z")
**Posts on this page:** 1
**Showing post:** 3

<div class="post-metadata">

### Author: ![gabriel\_tessier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabriel_tessier/32/27911_2.png) [@gabriel\_tessier](https://discuss.elastic.co/u/gabriel_tessier)
#### Post date: [June 12, 2020, 12:55am UTC](https://discuss.elastic.co/t/filebeat-lit-des-lignes-2-fois/236452/3 "2020-06-12T00:55:51Z")

</div>

Bonjour @IneedHelp,

Peux-tu partager un exemple de documents en double? Si ca viens du csv c'est que des lignes sont en double dans tes fichiers. Y'a des solutions pour deboublonner les lignes:

> **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)**
>
> Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output.

> [@Remove duplicates in logstash logs](https://discuss.elastic.co/t/remove-duplicates-in-logstash-logs/134057):
>
> Hello, I use filebeat to send logs to logstash and then I use Logstash to parse them. For exemple this line : 31/5/2018 01:06:24.073 (TACHE) 30/5/2018/D T UE\_TECXXX\_J\_TN\_SSH002\_DELCLONE\_LPSACAS1(14033)/UE\_TECXXX\_X\_LL\_OPE003\_TECHNIQUE TER STATUS : TN Terminaison normale de la tâche (TN EXIT CODE 0) Become after groke parse : { "DATE\_TACHE" =\> "30/5/2018" "HEURE\_TACHE" =\> "01:06:24.073" "NOM\_TACHE" =\> "UE\_TECXXX\_J\_TN\_SSH002\_DELCLONE\_LPSACAS1" "LOCALISATION\_TACHE" =\> "UE\_…

> [@Logstash -\> drop duplicate -\> elasticsearch](https://discuss.elastic.co/t/logstash-drop-duplicate-elasticsearch/77807/7):
>
> @Christian_Dahlqvist I see what your saying, so basically in order to detect a duplicate in logstash properly you'd have to create some kind of duplicate service and find a way to scale it which adds more complexity. Hmm. Ok, so as you said if Elasticsearch can handle it, basically a feature in elasticsearch to drop it would be more efficient. I will move my feature request from logstash to elasticsearch. Thanks, E

---

_[View the full topic](https://discuss.elastic.co/t/filebeat-lit-des-lignes-2-fois/236452)._