# Load multiple csv files and update existing index
**URL:** https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324
**Category:** Logstash
**Created:** [October 4, 2019, 10:16am UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324 "2019-10-04T10:16:17Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@myrtod](https://discuss.elastic.co/u/myrtod)
#### Post date: [October 4, 2019, 10:16am UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/1 "2019-10-04T10:16:17Z")
Hi, I am trying to create a "master table" in elastic, using logstash, where many csv files will be merged according to a common column named 'A'. Given the mapping of the index, can I insert each csv and update the unique column each csv refers to (col1, col2, col3, etc), by grouping regarding the common column 'A' ??
In more detail, I wish for each csv to append its info to the corresponding column (already existing) and if the group column 'A' value doesn't yet exist to insert it and then update the corresponding column, or else to just update the corresponding column for the given value of 'A'.
Any ideas ?? I have already tried merging the csv files using python but run out of memory (64GB).
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 4, 2019, 1:25pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/2 "2019-10-04T13:25:05Z")
It sounds like you want to use an elasticsearch output with [doc\_as\_upsert](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-doc_as_upsert) set, and use column 'A' as the document id.
---
### Author:  [@myrtod](https://discuss.elastic.co/u/myrtod)
#### Post date: [October 4, 2019, 1:40pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/3 "2019-10-04T13:40:44Z")
Yes exactly !
I have tried inserting the first csv, in the given mapping, and then insert a second one like that
`output{ elasticsearch{ hosts => ["localhost"] index => "mock" action => "update" doc_as_upsert => true }`
but I don't know how to use column 'A' as the doc\_id
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 4, 2019, 1:49pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/4 "2019-10-04T13:49:36Z")
> [@myrtod](#):
>
> I don't know how to use column 'A' as the doc\_id
Use the document\_id option and a sprintf reference to a field
```
document_id => "%{[A]}"
```
will use the contents of field A as the document id.
---
### Author:  [@myrtod](https://discuss.elastic.co/u/myrtod)
#### Post date: [October 4, 2019, 2:03pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/5 "2019-10-04T14:03:40Z")
Thank you ! That was really helpful !
One last question though... given that a csv has multiple data for a unique value of column A, can I append all of them somehow, or will the last one overwrite the previous one and thus end up with only 1 value per value of column A for each csv ?
---
### Author:  [@myrtod](https://discuss.elastic.co/u/myrtod)
#### Post date: [October 4, 2019, 2:11pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/7 "2019-10-04T14:11:19Z")
All csv files have the same number of columns (2) however for example, lets say csv\_1 is like this...
column\_A | Value
1 10
1 20
2 5
3 4
Would both values 10 and 20 be written for the unique value of column A being 1 or would only one of them survive, using doc\_as\_upsert =\> true and action =\> update ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 4, 2019, 2:27pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/8 "2019-10-04T14:27:24Z")
Only one would survive.
Another possible approach is shown [here](https://discuss.elastic.co/t/aggregate-problem/142572/7), which is to have logstash write out a file that contains document updates and then curl that into elasticsearch.
Yet another option would be to use a variant of [this](https://discuss.elastic.co/t/remove-fields-from-documents-based-on-their-value/200270/2). If an elasticsearch output does not do quite what you want then you can use an http filter to POST into elasticsearch.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 1, 2019, 2:38pm UTC](https://discuss.elastic.co/t/load-multiple-csv-files-and-update-existing-index/202324/9 "2019-11-01T14:38:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.