# How to read csv data in logstash and convert column to date,integer,boolean etc

**URL:** <https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056>\
**Category:** Logstash\
**Created:** [February 14, 2017, 2:47pm UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056 "2017-02-14T14:47:22Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [February 14, 2017, 2:47pm UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/1 "2017-02-14T14:47:22Z")

</div>

I got error when creating the index in logstash

Could not find log4j2 configuration at path /ElasticProducts/logstash-5.2.0/config/log4j2.properties. Using default config which logs to console  
08:38:21.769 [LogStash::Runner] ERROR logstash.agent - fetched an invalid config {:config=\>"input {\n\tstdin {\n\t\t\ttype =\> "stdin-type"\n\t\t}\n\tfile

config file as below

input {  
stdin {  
type =\> "stdin-type"  
}  
file {  
path =\> ["C:/Elastic/Data.csv"]  
start\_position =\> "beginning"  
}  
}

filter {  
csv {  
columns =\> ["ITIL\_Incident\_ID","Incident\_Number","Incident\_Create\_Date","Incident\_Created\_By\_Name","P1\_Incident\_Flag"]

separator =\> ","

convert =\> {

```
"ITIL_Incident_ID" => "integer" ,
"P1_Incident_Flag" => "boolean"

			}
}

```

date {  
match =\> ["Incident\_Create\_Date" , "mm/dd/yy HH:mm"]  
}

}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "csvfile"  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 15, 2017, 11:20pm UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/2 "2017-02-15T23:20:30Z")

</div>

Can you share the entire error?

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [February 16, 2017, 3:46am UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/3 "2017-02-16T03:46:37Z")

</div>

Could not find log4j2 configuration at path /ElasticProducts/logstash-5.2.0/config/log4j2.properties. Using default conf  
ig which logs to console  
21:40:56.393 [LogStash::Runner] ERROR logstash.agent - fetched an invalid config {:config=\>"input {\n\n\tstdin {\n\t\t\t  
path =\> ["C:/ElasticProducts/IncidentTestData.csv"]\n\t\t\tstart\_position =\> "beginning"\n\t\t}\n\t}\n\nfilter {\ncs  
v {\n\n\t\tcolumns =\> ["ITIL\_Incident\_ID","Incident\_Number","Incident\_Create\_Date","Incident\_Created\_By\_Name","  
P1\_Incident\_Flag"]\n\n\t\tseparator =\> ","\n\n\t\tconvert =\> { \n\n ["ITIL\_Incident\_ID" =\> "integer"]\n\t[   
"P1\_Incident\_Flag" =\> "integer" ]\n\n \t\t }\n\n\tdate {\n \t\tmatch =\> [ "Incident\_Create\_Date" , "mm/dd/yy  
HH:mm" ]\n \t\t}\n \t}\n\n}\n \n \n\noutput {\n elasticsearch {\n hosts =\> ["localhost:9200"]\n user =\> "e  
lastic"\n password =\> "changeme" \n # protocol =\> "http"\n index =\> "csvfile"\n #template =\> ""\n  
}\n # stdout { codec =\> rubydebug }\n}\n", :reason=\>"Expected one of #, -, ", ', } at line 18, column 5 (byte 294) afte  
r filter {\ncsv {\n\n\t\tcolumns =\> ["ITIL\_Incident\_ID","Incident\_Number","Incident\_Create\_Date","Incident\_Create  
d\_By\_Name","P1\_Incident\_Flag"]\n\n\t\tseparator =\> ","\n\n\t\tconvert =\> { \n\n "}

New config file is

input {

```
stdin {
		path => ["C:/ElasticProducts/IncidentTestData.csv"]
		start_position => "beginning"
	}
}

```

filter {  
csv {

```
	columns => ["ITIL_Incident_ID","Incident_Number","Incident_Create_Date","Incident_Created_By_Name","P1_Incident_Flag"]

	separator => ","
	convert => { 

["ITIL_Incident_ID" => "integer"]
["P1_Incident_Flag" => "integer"]

		 }

date {
		match => ["Incident_Create_Date" , "mm/dd/yy HH:mm"]
	}
}

```

}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
user =\> "elastic"  
password =\> "changeme"  
# protocol =\> "http"  
index =\> "csvfile"  
#template =\> ""  
}

# stdout { codec =\> rubydebug }

}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 16, 2017, 4:28am UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/4 "2017-02-16T04:28:34Z")

</div>

The convert syntax is wrong, check out [https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-convert](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-convert)

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [February 16, 2017, 4:49am UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/5 "2017-02-16T04:49:28Z")

</div>

i still got error even after changing the convert format

PS C:\ElasticProducts\logstash-5.2.0\bin\> logstash -f csv.conf

Could not find log4j2 configuration at path /ElasticProducts/logstash-5.2.0/config/log4j2.properties. Using default config which logs to console  
22:44:58.945 [LogStash::Runner] ERROR logstash.agent - fetched an invalid config {:config=\>"input {\n\n\tfile {\n\t\t\tpath =\> ["C:/ElasticProducts/IncidentTestData.csv"]\n\t\t\tstart\_positi  
on =\> "beginning"\n\t\t\tsincedb\_path =\> "/dev/null"\n\t\t}\n\t}\n\nfilter {\ncsv {\n\n\t\tcolumns =\> ["ITIL\_Incident\_ID","Incident\_Number","Incident\_Create\_Date","Incident\_Created\_  
By\_Name","P1\_Incident\_Flag"]\n\n\t\tseparator =\> ","\n\n\tconvert =\> { "ITIL\_Incident\_ID" =\> "integer" , "P1\_Incident\_Flag" =\> "boolean" }\n\n \t}\n\n \tdate {\n \t\tmatch =

> ["Incident\_Create\_Date" , "mm/dd/yy HH:mm"]\n \t }\n}\n \n \n\noutput {\n elasticsearch {\n hosts =\> ["localhost:9200"]\n user =\> "elastic"\n password =\> "changeme" \n  
> # protocol =\> "http"\n index =\> "csvfile"\n #template =\> ""\n }\n stdout {\n id =\> "ITIL\_Incident\_ID"\n }\n}\n", :reason=\>"Expected one of #, {, -, ", ', } at line 17, c  
> olumn 47 (byte 349) after filter {\ncsv {\n\n\t\tcolumns =\> ["ITIL\_Incident\_ID","Incident\_Number","Incident\_Create\_Date","Incident\_Created\_By\_Name","P1\_Incident\_Flag"]\n\n\t\tseparat  
> or =\> ","\n\n\tconvert =\> { "ITIL\_Incident\_ID" =\> "integer" "}

config file is as below

filter {  
csv {

```
	columns => ["ITIL_Incident_ID","Incident_Number","Incident_Create_Date","Incident_Created_By_Name","P1_Incident_Flag"]
	separator => ","

convert => { "ITIL_Incident_ID" => "integer" , "P1_Incident_Flag" => "integer" }

}

date {
		match => ["Incident_Create_Date" , "mm/dd/yy HH:mm"]
	 }

```

}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 16, 2017, 4:58am UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/6 "2017-02-16T04:58:13Z")

</div>

Please make sure you provide the entire error!

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [February 16, 2017, 5:04am UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/7 "2017-02-16T05:04:50Z")

</div>

it has full error message. it saying error in line 17 column 47 now

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 16, 2017, 7:00am UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/8 "2017-02-16T07:00:37Z")

</div>

There shouldn't be any commas between the hash items, i.e.

```
convert => { "ITIL_Incident_ID" => "integer" , "P1_Incident_Flag" => "integer" }

```

should be

```
convert => { "ITIL_Incident_ID" => "integer" "P1_Incident_Flag" => "integer" }

```

This is a documentation bug that was fixed a long time ago but a new plugin release hasn't been made so [elastic.co](http://elastic.co) is out of date.

---

<div class="post-metadata">

**Author:** ![varun1992](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@varun1992](https://discuss.elastic.co/u/varun1992)\
**Post date:** [February 16, 2017, 12:35pm UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/9 "2017-02-16T12:35:47Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/2X/6/6b270c1ca492cdca9da644b424e3f8960cdb7c09.png)

Index created successfully, But still datatype is string and there is multiple fields created for same field. Eg : Incident\_Create\_Date and Incident\_Create\_Date.keyword

Why is that ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 16, 2017, 9:45pm UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/10 "2017-02-16T21:45:35Z")

</div>

We don't know what you want to do with the mappings, so we index the fields as text and keyword.

A keyword is [https://www.elastic.co/guide/en/elasticsearch/reference/5.2/keyword.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/keyword.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2017, 9:45pm UTC](https://discuss.elastic.co/t/how-to-read-csv-data-in-logstash-and-convert-column-to-date-integer-boolean-etc/75056/11 "2017-03-16T21:45:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.