# Logstash ingested CSV file twice **URL:** \ **Category:** Logstash\ **Created:** [June 18, 2019, 11:06pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351 "2019-06-18T23:06:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![victoravr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/victoravr/32/39901_2.png) [@victoravr](https://discuss.elastic.co/u/victoravr)\ **Post date:** [June 18, 2019, 11:06pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351/1 "2019-06-18T23:06:47Z")
For some reason Logstash processed CSV file twice. Could you please help to find out why? Running logstash.version 6.5.3. input { file { path =\> "/path/to/\*.csv" start\_position =\> "beginning" mode =\> "read" file\_completed\_action =\> "log" file\_completed\_log\_path =\> "/path/to/logs/processed.log" } File "/path/to/logs/processed.log" contains two records of "/path/to/input.csv" when it should be only one! Also, there are double amount of documents in ElasticSearch with duplicates from this file. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 18, 2019, 11:15pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351/2 "2019-06-18T23:15:18Z")
The sincedb should prevent this. You would have to provide more information, including matching sincedb entries and entries from the file completed log. Did you restart logstash? ---
**Author:** ![victoravr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/victoravr/32/39901_2.png) [@victoravr](https://discuss.elastic.co/u/victoravr)\ **Post date:** [June 18, 2019, 11:32pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351/3 "2019-06-18T23:32:42Z")
Yes, sincedb should've prevented this. There are two entries of "/path/to/input.csv" in "/path/to/logs/processed.log". And there is one entry for "/path/to/input.csv" in /var/lib/logstash/plugins/inputs/file/.sincedb\_ like this: 4278179 0 64769 758 1560832674.226105 "/path/to/input.csv" No, I haven't restarted logstash. ---
**Author:** ![victoravr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/victoravr/32/39901_2.png) [@victoravr](https://discuss.elastic.co/u/victoravr)\ **Post date:** [June 19, 2019, 4:28am UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351/4 "2019-06-19T04:28:10Z")
Well, just found out that logstash reads CSV file each time it restarts! it recreates .sincedb file upon start and processing file from the beginning. Is there a bug in Logstash that has been fixed in later versions? Or is it "by design"? Now I'm thinking of a workaround - to write a bash script that moves files after certain time to a different directory (not to loose them). ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 19, 2019, 12:47pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351/5 "2019-06-19T12:47:58Z")
> [@victoravr](#): > > it recreates .sincedb file upon start and processing file from the beginning. > > Is there a bug in Logstash that has been fixed in later versions? Or is it "by design"? It's certainly not "by design". The only reason to have a sincedb on disk is to persist the in-memory sincedb across restarts so that the file input does not re-read files when logstash is restarted. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2019, 12:48pm UTC](https://discuss.elastic.co/t/logstash-ingested-csv-file-twice/186351/6 "2019-07-17T12:48:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.