# Can't parse a CSV file - message is coming through though **URL:** \ **Category:** Logstash\ **Created:** [August 12, 2017, 6:18pm UTC](https://discuss.elastic.co/t/cant-parse-a-csv-file-message-is-coming-through-though/96855 "2017-08-12T18:18:56Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [August 12, 2017, 6:18pm UTC](https://discuss.elastic.co/t/cant-parse-a-csv-file-message-is-coming-through-though/96855/1 "2017-08-12T18:18:56Z")
- update - problem solved, explanations in the replies section Hi, I am using the conf file below to parse a csv. I cannot get the message to populate into the fields that are mapped. Instead I see the message in kibana, but the fields are all zeros. stdout shows the fields are all parsed. After 3 days of trying this out I could use some help. Kibana screenshot result is here showing the message is going through ok: [[Album] Imgur: The magic of the Internet ![](https://i.imgur.com/UhNH1Xo.jpg?fb "Imgur: The magic of the Internet") ](https://imgur.com/a/UQSYL) You should also know I removed a mutate clause to tell logstash what is an integer/float, but even then the parsing does not map into fields. input { file { path =\> ["/home/elastic/turbofiles/dxc/_.csv"] # for future use: # recursive search under path # path =\> ["/home/elastic/turbofiles/dxc/\*\*/_.csv"] ``` start_position => beginning sincedb_path => "/dev/null" ``` } } filter { csv { #source =\> "message" separator =\> ";" skip\_empty\_columns =\> true quote\_char =\> "'" columns =\> ["run\_date", "location\_continent", "location\_state\_country", "location\_city", "clusterUuid", "VMs", "Hosts", "VM\_Headroom", "VMHost\_Density", "AvgCPUUsed", "UtilCPU", "PeakCPUUsed", "AvgCPUCapEff", "AvgCPUCapHA", "AvgMemUsed", "UtilMem", "PeakMemUsed", "AvgMemCapEff", "MaxMemCapHA", "DS\_AvgSA", "DS\_AvgSAFree", "DS\_UtilSA", "DS\_AvgSACap", "DS\_AvgSP", "DS\_AvgSPFree", "DS\_UtilSP", "DS\_AvgSPCap", "VM\_AvgVCPU", "VM\_UtilVCPU", "VM\_PeakVCPU", "VM\_AvgVMem", "VM\_UtilVMem", "VM\_PeakVMem", "VM\_AvgIO", "VM\_UtilIO", "VM\_PeakIO", "VM\_AvgSL", "VM\_UtilSL", "VM\_PeakSL", "VM\_AvgSA", "VM\_UtilSA", "VM\_PeakSA", "VM\_AvgSP", "VM\_UtilSP", "VM\_PeakSP", "Alloc\_VM\_VMem", "Alloc\_Mem\_Ratio", "Alloc\_Mem\_%", "Alloc\_VM\_VCPU", "Alloc\_CPU\_Ratio", "Alloc\_CPU\_%"] } if [VMs] == "VMs" { drop { } } } filter { grok { match =\> ["path", "\_%{YEAR:year}%{MONTHNUM:month}%{MONTHDAY:day}.csv$"] add\_field =\> ["datetime", "%{year}.%{month}.%{day}"] add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] } mutate { remove\_field =\> ["month", "day", "year"] } } output { elasticsearch { hosts =\> "localhost" index =\> "test" document\_type =\> "test\_report" } stdout { } } Example for a csv file "as is" with the header line, which is parsed even when I tried to filter it out. "run\_date";"location\_continent";"location\_state\_country";"location\_city";"clusterUuid";"VMs";"Hosts";"VM\_Headroom";"VMHost\_Density";"AvgCPUUsed";"UtilCPU";"PeakCPUUsed";"AvgCPUCapEff";"AvgCPUCapHA";"AvgMemUsed";"UtilMem";"PeakMemUsed";"AvgMemCapEff";"MaxMemCapHA";"DS\_AvgSA";"DS\_AvgSAFree";"DS\_UtilSA";"DS\_AvgSACap";"DS\_AvgSP";"DS\_AvgSPFree";"DS\_UtilSP";"DS\_AvgSPCap";"VM\_AvgVCPU";"VM\_UtilVCPU";"VM\_PeakVCPU";"VM\_AvgVMem";"VM\_UtilVMem";"VM\_PeakVMem";"VM\_AvgIO";"VM\_UtilIO";"VM\_PeakIO";"VM\_AvgSL";"VM\_UtilSL";"VM\_PeakSL";"VM\_AvgSA";"VM\_UtilSA";"VM\_PeakSA";"VM\_AvgSP";"VM\_UtilSP";"VM\_PeakSP";"Alloc\_VM\_VMem";"Alloc\_Mem\_Ratio";"Alloc\_Mem\_%";"Alloc\_VM\_VCPU";"Alloc\_CPU\_Ratio";"Alloc\_CPU\_%" "2017-08-15";"NA";"NY";"NEWYORK";"16d994b166b78a80e7f461f52470";"11";"4";"20";"2.75";"4.15";"15.96";"4.71";"26.03";"0";"8.90";"44.51";"3.93";"20.00";"0";"3436.70";"2652.71";"56.44";"6089.42";"5102.45";"7076.38";"41.90";"12178.83";"2.45";"7.83";"6.31";"0.19";"2.76";"0.96";"N;"N;"N;"60.44";"60.44";"3532.00";"12.13";"0.04";"12.46";"71.14";"0.12";"71.48";"7.00";"0.35";"35.01";"31.24";"1.20";"120.00" "2017-08-15";"NA";"NY";"NEWYORK";"afb62dd5426e7a3891747a11ed4b";"29";"4";"63";"7.25";"7.46";"14.34";"4.09";"52.06";"0";"24.41";"30.51";"8.29";"80.00";"0";"3436.02";"2653.40";"56.43";"6089.42";"5176.45";"7002.39";"42.50";"12178.83";"4.89";"4.72";"7.89";"0.81";"1.33";"2.33";"N;"N;"N;"49.58";"49.58";"2919.00";"366.19";"0.70";"366.19";"1366.45";"1.31";"1366.45";"60.60";"0.76";"75.76";"103.52";"1.99";"198.84" ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [August 13, 2017, 12:02am UTC](https://discuss.elastic.co/t/cant-parse-a-csv-file-message-is-coming-through-though/96855/2 "2017-08-13T00:02:35Z")
OK update - the mutate/convert is killing my values. Not sure why, I tried to play with integer vs. float A number like 17.23 converts into zero no matter what I choose (float/integer) Also - I wish there was a "live debug" option for logstash. To step into each field and see what logstash wants to do and perhaps play with the real-time settings to see what logstash will output. Just an idea. ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [August 14, 2017, 4:41am UTC](https://discuss.elastic.co/t/cant-parse-a-csv-file-message-is-coming-through-though/96855/3 "2017-08-14T04:41:03Z")
The problem is now solved, the main issue I had with "mutate" not working is that I should have left the seperator char as the default. This way all the data surrounded by double quotes was treated as expected with the help of "mutate... convert" Also - my if/drop now works, and the date filter works for my "run\_date" field. Cheers, JD ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 11, 2017, 4:41am UTC](https://discuss.elastic.co/t/cant-parse-a-csv-file-message-is-coming-through-though/96855/4 "2017-09-11T04:41:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.