Hi, I'm wondering if anyone has a good solution to the following. I have to read in .csv files that are generated every hour. These .csv files contain all the historic data for as far back as the program has run. Every hour it runs again and creates a new file that may contain new data at the end,…

Preventing duplicates when reading the same data multiple times

wlbsxnlp22 May 25, 2021, 8:34am 3

Thank you! That did indeed work.

Topic		Replies	Views
Reg csv file import into Elasticsearch using Logstash Logstash	13	2003	February 26, 2021
How to remove duplicate events in logstash Logstash	3	4419	January 4, 2017
Duplicate Entries of Log data Elasticsearch	6	4801	September 29, 2017
Avoid duplication Logstash	13	5203	December 7, 2018
Keeping duplicates Logstash	5	281	September 19, 2019

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Preventing duplicates when reading the same data multiple times

Related topics