Filebeat lit des lignes 2 fois

[gabriel_tessier]

Bonjour @IneedHelp,

Peux-tu partager un exemple de documents en double? Si ca viens du csv c'est que des lignes sont en double dans tes fichiers. Y'a des solutions pour deboublonner les lignes:

Little Logstash Lessons: Handling Duplicates

Remove duplicates in logstash logs

Hello, I use filebeat to send logs to logstash and then I use Logstash to parse them. For exemple this line : 31/5/2018 01:06:24.073 (TACHE) 30/5/2018/D T UE_TECXXX_J_TN_SSH002_DELCLONE_LPSACAS1(14033)/UE_TECXXX_X_LL_OPE003_TECHNIQUE TER STATUS : TN Terminaison normale de la tâche (TN EXIT CODE 0) Become after groke parse : { "DATE_TACHE" => "30/5/2018" "HEURE_TACHE" => "01:06:24.073" "NOM_TACHE" => "UE_TECXXX_J_TN_SSH002_DELCLONE_LPSACAS1" "LOCALISATION_TACHE" => "UE_…

Logstash -> drop duplicate -> elasticsearch

@Christian_Dahlqvist I see what your saying, so basically in order to detect a duplicate in logstash properly you'd have to create some kind of duplicate service and find a way to scale it which adds more complexity. Hmm. Ok, so as you said if Elasticsearch can handle it, basically a feature in elasticsearch to drop it would be more efficient. I will move my feature request from logstash to elasticsearch. Thanks, E