# CSV file indexed to %{\[@metadata\] instead of target
**URL:** https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957
**Category:** Logstash
**Created:** [August 4, 2017, 10:27pm UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957 "2017-08-04T22:27:40Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@bulsac](https://discuss.elastic.co/u/bulsac)
#### Post date: [August 4, 2017, 10:27pm UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/1 "2017-08-04T22:27:40Z")
I am attempting to ingest a CSV file using logstash into an elasticsearch index that I have pre-defined using a PUT request. It appears that logstash is ingesting the CSV file not into the intended elasticsearch index, but into one labelled:
%{[@metadata][beat]}-2017.08.04
I am pasting the relevant logstash conf file, as well as the PUT request I used to pre-build the index. High-fives for any help:
**BEGIN LOGSTASH CONF**
```
input {
file {
path => "/var/elk/csv/sep/*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Pattern_Date","Operating_System","Client_Version","Policy_Serial",
"HI_Status","Status","Auto_Protect_On","Worst_Detection",
"Last_Scan_Time","Antivirus_engine_On","Download_Insight_On",
"SONAR_On","Tamper_Protection_On","Intrusion_Prevention_On",
"IE_Browser_Protection_On","Firefox_Browser_Protection_On",
"Early_Launch_Antimalware_On","Computer_Name","Server_Name",
"MAC_Address1"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "sep-index"
}
}
```
**END CONF**
**BEGIN INDEX CREATION**
PUT sep-index
{
"mappings": {
"logs": {
"properties": {
"@timestamp": {
"type": "date",
"format": "basic\_date"
},
"@version": {
"type": "string"
},
"Pattern\_Date": {
"type": "date",
"format": "basic\_date"
},
"Operating\_System": {
"type": "keyword"
},
"Policy\_Serial": {
"type": "keyword"
},
"HI\_Status": {
"type": "keyword"
},
"Status": {
"type": "keyword"
},
"Auto\_Protect\_On": {
"type": "keyword"
},
"Worst\_Detection": {
"type": "keyword"
},
"Last\_Scan\_Time": {
"type": "date",
"format": "basic\_date"
},
"Antivirus\_engine\_On": {
"type": "keyword"
},
"Download\_Insight\_On": {
"type": "keyword"
},
"SONAR\_On": {
"type": "keyword"
},
"Tamper\_Protection\_On": {
"type": "keyword"
},
"Intrusion\_Prevention\_On": {
"type": "keyword"
},
"IE\_Browser\_Protection\_On": {
"type": "keyword"
},
"Firefox\_Browser\_Protection\_On": {
"type": "keyword"
},
"Early\_Launch\_Antimalware\_On": {
"type": "keyword"
},
"Computer\_Name": {
"type": "keyword"
},
"Server\_Name": {
"type": "keyword"
},
"MAC\_Address1": {
"type": "keyword"
}
}
}
}
}
**END INDEX CREATION**
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [August 5, 2017, 2:01am UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/2 "2017-08-05T02:01:17Z")
Please format your code using `` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21). It will make your post more readable.
Or use markdown style like:
````
```
CODE
```
````
I moved your question to #logstash
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 5, 2017, 3:33pm UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/3 "2017-08-05T15:33:20Z")
> It appears that logstash is ingesting the CSV file not into the intended elasticsearch index, but into one labelled:
>
> %{[@metadata][beat]}-2017.08.04
That's because you have
```
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
```
in one of your configuration files but your CSV events don't have a `[@metadata][beat]` field. Multiple configuration files are effectively concatenated so you need to use conditionals if you don't want all events to be sent to all outputs.
---
### Author:  [@bulsac](https://discuss.elastic.co/u/bulsac)
#### Post date: [August 6, 2017, 4:40am UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/4 "2017-08-06T04:40:34Z")
Thank you Magnus. I found and addressed the issue in one of my other conf files. I still cannot get data from the csv file into the desired index, but I now know that it has something to do with how I pre-built the index. I know this because when I do not pre-build the index, the csv file is ingested as intended
I'm not sure what I did wrong with the PUT, or if perhaps I missed a step after it. Any help is appreciated. Re-pasting the PUT I posted using Kibana development console:
```
PUT sep-index
{
“mappings”: {
“logs”: {
“properties”: {
"@timestamp": {
“type”: “date”,
“format”: “basic_date”
},
"@version": {
“type”: “string”
},
“Pattern_Date”: {
“type”: “date”,
“format”: “basic_date”
},
“Operating_System”: {
“type”: “keyword”
},
“Policy_Serial”: {
“type”: “keyword”
},
“HI_Status”: {
“type”: “keyword”
},
“Status”: {
“type”: “keyword”
},
“Auto_Protect_On”: {
“type”: “keyword”
},
“Worst_Detection”: {
“type”: “keyword”
},
“Last_Scan_Time”: {
“type”: “date”,
“format”: “basic_date”
},
“Antivirus_engine_On”: {
“type”: “keyword”
},
“Download_Insight_On”: {
“type”: “keyword”
},
“SONAR_On”: {
“type”: “keyword”
},
“Tamper_Protection_On”: {
“type”: “keyword”
},
“Intrusion_Prevention_On”: {
“type”: “keyword”
},
“IE_Browser_Protection_On”: {
“type”: “keyword”
},
“Firefox_Browser_Protection_On”: {
“type”: “keyword”
},
“Early_Launch_Antimalware_On”: {
“type”: “keyword”
},
“Computer_Name”: {
“type”: “keyword”
},
“Server_Name”: {
“type”: “keyword”
},
“MAC_Address1”: {
“type”: “keyword”
}
}
}
}
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 6, 2017, 6:50pm UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/5 "2017-08-06T18:50:54Z")
Is Logstash even reading the CSV file? Have you looked in the Logstash log for clues?
---
### Author:  [@bulsac](https://discuss.elastic.co/u/bulsac)
#### Post date: [August 7, 2017, 4:33am UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/6 "2017-08-07T04:33:11Z")
Nothing in the logstash logs. However, when I look at my elasticsearch logs I see a huge number of entries that look like this:
`[DEBUG][o.e.a.b.TransportShardBulkAction] [3NZmxIH] [sep-index][4] failed to execute bulk item (index) BulkShardRequest [[sep-index][4]] containing [4] requests`
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 7, 2017, 5:15am UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/7 "2017-08-07T05:15:34Z")
That doesn't look normal. Any other interesting log entries? Is the cluster's health green (or at least yellow)?
---
### Author:  [@bulsac](https://discuss.elastic.co/u/bulsac)
#### Post date: [August 8, 2017, 5:18am UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/8 "2017-08-08T05:18:41Z")
When I grep'd the logs a different way I found that elasticsearch was failing to parse my @timestamp field. When I removed "format": "basic\_date" from the field properties everything started working.
Thank you for your help!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 5, 2017, 5:18am UTC](https://discuss.elastic.co/t/csv-file-indexed-to-metadata-instead-of-target/95957/9 "2017-09-05T05:18:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.