# Problem beetween elastic and logstash: csv files are saved twice

**URL:** <https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655>\
**Category:** Logstash\
**Created:** [June 11, 2020, 9:07am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655 "2020-06-11T09:07:11Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![IneedHelp](https://avatars.discourse-cdn.com/v4/letter/i/eada6e/32.png) [@IneedHelp](https://discuss.elastic.co/u/IneedHelp)\
**Post date:** [June 11, 2020, 9:07am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/1 "2020-06-11T09:07:11Z")

</div>

Hello !  
I use ELK to parse log file and csv file but yesterday a little problem appeared. All the data in my csv file were saved twice so a 1000 lines document is now a 2000 lines document. In fact you just have to divide all the result by two but my dashboards are on the local network so it's annoying for the others users.

First of all, I tried to locate the problem so i changed the output of my logstash config file with stdout{} and they were no problem with the output. Therefore I think the problem is beetween elastic and logstash.

I cheked elasticsearch but didn't find anything. I use the same logstash config file to parse log and csv so i don't understand while only csv files are impacted.

Here my logstash config file:

```auto
input { 
	beats {
        port => "5044"
    }
}
filter {
       if "Log" in [tags] {
       ...
       }
       if "Csv" in [tags] {
       ...
       }
}
output {
	elasticsearch {
		hosts => ["127.0.0.1:9200"]
		index => "squid-%{File_Type}"
	}
}  

```

( File\_Type is csv or log )

If you any idea i take it !  
Thanks you

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 12, 2020, 3:48am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/2 "2020-06-12T03:48:12Z")

</div>

To make sure it does not happen, you should use one the columns of your CSV file as the `_id` of the document. That way, if for whatever reason the file gets parsed again, you will just overwrite the existing values.

---

<div class="post-metadata">

**Author:** ![Thompso1n](https://avatars.discourse-cdn.com/v4/letter/t/f14d63/32.png) [@Thompso1n](https://discuss.elastic.co/u/Thompso1n)\
**Post date:** [June 12, 2020, 6:02am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/3 "2020-06-12T06:02:18Z")

</div>

This Logstash configuration file directs Logstash to read apache error logs[!](https://www.dqfansurvey.org/dqfanfeedback/)

---

<div class="post-metadata">

**Author:** ![IneedHelp](https://avatars.discourse-cdn.com/v4/letter/i/eada6e/32.png) [@IneedHelp](https://discuss.elastic.co/u/IneedHelp)\
**Post date:** [June 22, 2020, 7:23am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/4 "2020-06-22T07:23:52Z")

</div>

I'm sorry for this delayed answer but i took a vacation 🕶  
Thanks for your answer, i read a similar answer 1 week ago but i don't know how to do it. Can you exlain me please ?  
In fact, Should I use the line number as the `_id` or add an id column in all my csv files and define it as the `_id` ?

---

<div class="post-metadata">

**Author:** ![IneedHelp](https://avatars.discourse-cdn.com/v4/letter/i/eada6e/32.png) [@IneedHelp](https://discuss.elastic.co/u/IneedHelp)\
**Post date:** [June 22, 2020, 7:26am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/5 "2020-06-22T07:26:04Z")

</div>

Like for daddonet I'm sorry for the delayed answer but i thanks you for your answer.  
What do you mean by:

> read apache error logs

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 22, 2020, 8:27am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/6 "2020-06-22T08:27:13Z")

</div>

Both would work I guess but the easiest is to add a column. Note that logstash won't be able to know the line number I think.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 20, 2020, 8:27am UTC](https://discuss.elastic.co/t/problem-beetween-elastic-and-logstash-csv-files-are-saved-twice/236655/7 "2020-07-20T08:27:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.