# Multiple .csv file data is being added together

**URL:** <https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145>\
**Category:** Logstash\
**Created:** [September 12, 2017, 12:31am UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145 "2017-09-12T00:31:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cxbarbosa](https://avatars.discourse-cdn.com/v4/letter/c/df705f/32.png) [@cxbarbosa](https://discuss.elastic.co/u/cxbarbosa)\
**Post date:** [September 12, 2017, 12:31am UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145/1 "2017-09-12T00:31:16Z")

</div>

I have a wildcard to ingest multiple .csv files, however in the elastic search visualizations, its adding the same column items together instead of skipping repeat entries.

I want to create a data dump where new logs update only new items.

I'm pretty sure I read by default it would do this.  
Is there a setting in the filter plugin I need to add?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 12, 2017, 1:23am UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145/2 "2017-09-12T01:23:42Z")

</div>

> [@cxbarbosa](#):
>
> its adding the same column items together instead of skipping repeat entries.

Can you clarify what you mean by that?

---

<div class="post-metadata">

**Author:** ![cxbarbosa](https://avatars.discourse-cdn.com/v4/letter/c/df705f/32.png) [@cxbarbosa](https://discuss.elastic.co/u/cxbarbosa)\
**Post date:** [September 12, 2017, 4:55pm UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145/3 "2017-09-12T16:55:59Z")

</div>

I have user information entries on .csv files being downloaded as reports

I want to be able to have those reports sent automatically into log stash, adding only new records.

currently when I add a new report, it sees all the entries as brand new ones, and adds them together.

so if I have 30,000 items in a column, when I add a near identical report to the log stash folder, it then changes it to 60,000 and so on.

It was to my understanding that log stash could recognize this and only update new entries.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 12, 2017, 10:45pm UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145/4 "2017-09-12T22:45:52Z")

</div>

> [@cxbarbosa](#):
>
> It was to my understanding that log stash could recognize this and only update new entries.

Nope, Logstash is not a state machine and treats everything is a unique event.

You may want to look at creating a unique [document ID](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id) based on some of the fields. That way if you download another file that has the same entries, Logstash will simply update the existing one in Elasticsearch and not create a new one.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2017, 10:50pm UTC](https://discuss.elastic.co/t/multiple-csv-file-data-is-being-added-together/100145/5 "2017-10-10T22:50:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.