# Indexing csv data in a nested field using logstash

**URL:** <https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912>\
**Category:** Logstash\
**Created:** [May 12, 2016, 2:58pm UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912 "2016-05-12T14:58:20Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tamil](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@tamil](https://discuss.elastic.co/u/tamil)\
**Post date:** [May 12, 2016, 2:58pm UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912/1 "2016-05-12T14:58:20Z")

</div>

With below settings im unable to index the data .  
Logstash pipeline starts and stands still without showing any error or producing output.  
Please correct me where it is going wrong

**Input csv:**

cust\_name,state1,city1,state2,city2  
ab,CA,LA,IL,Chicago

**Mapping used :**

"mappings": {  
"info": {  
"properties": {  
"cust\_name": {  
"type": "string"  
},  
"address": {  
"properties": {  
"address1": {  
"properties": {  
"city": {  
"type": "string"  
},  
"state": {  
"type": "string"  
}  
}  
},  
"address2": {  
"properties": {  
"city": {  
"type": "string"  
},  
"state": {  
"type": "string"  
}  
}  
}  
},  
"type": "nested"  
}  
}  
}  
}

**conf file :**

input {  
file {  
path =\> "/home/cloudera/Desktop/nested\_csv.csv"  
type =\> "core2"  
start\_position =\> "beginning"  
}  
}  
filter {  
csv {  
columns =\> ["cust\_name", "state1","city1","state2","city2"]  
separator =\> ","  
}  
mutate{  
rename =\> {  
"state1" =\> "[address][address1][state]"  
"city1" =\> "[address][address1][city]"  
"state2" =\> "[address][address2][state]"  
"city2" =\> "[address][address2][city]"  
}  
}  
}  
output {  
elasticsearch {  
action =\> "index"  
hosts =\> ["localhost:9200"]  
index =\> "nested\_sample"  
document\_type =\> "info"  
workers =\> 1  
}  
}

**Desired output :**  
cust\_name: "ab",  
address : {  
address1 : {  
state:"CA",  
city:"LA"  
},  
address2:{  
state:"IL",  
city:"Chicago"  
}  
}

Thanks in Advance.

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [May 12, 2016, 8:44pm UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912/2 "2016-05-12T20:44:24Z")

</div>

I just tried your filter config, and it works great !

I think your problem is that you have a "sincedb" file pointing to the end of your csv file, so csv content is not processed.

So I invite you to stop logstash, delete all files `$HOME/.sincedb*` and restart logstash.

---

<div class="post-metadata">

**Author:** ![tamil](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@tamil](https://discuss.elastic.co/u/tamil)\
**Post date:** [May 13, 2016, 6:09am UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912/3 "2016-05-13T06:09:19Z")

</div>

Yes it works.. Thank you Fab for your response.  
Do i have to do the above said .sincedb file deletion every time ?  
Is there a workaround to avoid this scenario?

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [May 13, 2016, 7:45am UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912/4 "2016-05-13T07:45:19Z")

</div>

Normally, you have just to do it one time, given that you have a program which writes into input csv  
file or you have input file configured with path containing `*` and each time a new csv file is detected, logstash will process it from beginning.

---

<div class="post-metadata">

**Author:** ![Tribbles](https://avatars.discourse-cdn.com/v4/letter/t/9fc348/32.png) [@Tribbles](https://discuss.elastic.co/u/Tribbles)\
**Post date:** [May 13, 2016, 8:44am UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912/5 "2016-05-13T08:44:32Z")

</div>

Other than that, if you need to make sure to index the CSV, no matter what, you can always set the sincedb\_path in the input to /dev/null and LS will read in the file again whenever needed.

Thorsten

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:57am UTC](https://discuss.elastic.co/t/indexing-csv-data-in-a-nested-field-using-logstash/49912/6 "2017-07-06T04:57:44Z")

</div>