# Problem with csv filter

**URL:** <https://discuss.elastic.co/t/problem-with-csv-filter/233880>\
**Category:** Logstash\
**Created:** [May 22, 2020, 10:16am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880 "2020-05-22T10:16:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\
**Post date:** [May 22, 2020, 10:16am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/1 "2020-05-22T10:16:47Z")

</div>

Hi I have a csv file, in order to index it in elasticsearch I am using logstash. I am using CSV filter and my conf file is:  
'''input {  
file {  
sincedb\_path =\> "/null"  
path =\> "/home/kagamee/Downloads/time\_logs.csv"  
start\_position =\> "beginning"  
type =\> "logs"  
}  
}  
filter {  
csv{  
separator =\> ","  
columns =\> [  
"nil",  
"date",  
"time",  
"logid",  
"type",  
"subtype",  
"level",  
"vd",  
"eventtime",  
"srcip",  
"srcport",  
"srcintf",  
"srcintfrole",  
"dstip",  
"dstport",  
"dstintf",  
"dstintfrole",  
"action",  
"policyid",  
"policytype",  
"service",  
"dstcountry",  
"srccountry",  
"tradisp",  
"sentpkt"  
]  
skip\_empty\_columns =\> true  
quote\_char =\> "'"  
}  
}  
output {  
stdout { codec =\> rubydebug }  
#elasticsearch {

# hosts =\> ["localhost:9200"]

# index =\> "sample"

# }

}'''  
but after parsing the data I am getting output as:  
'''{  
"host" =\> "kagamee-Lenovo-Legion-Y7000P-1060",  
"srccountry" =\> """" srccountry=""Russian Feder"",  
"policytype" =\> ""5 policytype=""policy"",  
"vd" =\> ""vd=""root"""",  
"tradisp" =\> ""ation"" trandisp=""noop"" duration=0 sentbyte=0 rcvd"",  
"sentpkt" =\> ""byte=0 sentpkt=0 appcat=""unscanned"" crscore=30 craction=131072 crlevel=""high"""",  
"date" =\> "date=2020-05-18",  
"logid" =\> ""logid=""0000000013"""",  
"eventtime" =\> "eventtime=1589824586",  
"@timestamp" =\> 2020-05-22T10:07:24.656Z,  
"action" =\> ""6 action=""deny"",  
"path" =\> "/home/kagamee/Downloads/time\_logs.csv",  
"type" =\> ""type=""traffic"""",  
"srcintf" =\> ""srcintf=""port1"",  
"dstcountry" =\> """" dstcountry=""India"",  
"level" =\> ""level=""notice"""",  
"subtype" =\> ""subtype=""forward"""",  
"srcintfrole" =\> """" srcintfrole=""undefined"",  
"@version" =\> "1",  
"message" =\> ",date=2020-05-18,time=23:26:26,"logid=""0000000013""","type=""traffic""","subtype=""forward""","level=""notice""","vd=""root""",eventtime=1589824586,srcip=178.154.200.94,srcport=41798,"srcintf=""port1",""" srcintfrole=""undefined",""" dstip=45.249.108.19",7 dstport=44,"3 dstintf=""port2",""" dstintfrole=""undefined"" poluuid=""9bd3b56e-98c4-51ea-b15c-b04fcdf1a572"" sessionid=1629401503 proto=","6 action=""deny",""" policyid=3","5 policytype=""policy",""" service=""HTTPS",""" dstcountry=""India",""" srccountry=""Russian Feder","ation"" trandisp=""noop"" duration=0 sentbyte=0 rcvd","byte=0 sentpkt=0 appcat=""unscanned"" crscore=30 craction=131072 crlevel=""high"""",  
"dstintfrole" =\> """" dstintfrole=""undefined"" poluuid=""9bd3b56e-98c4-51ea-b15c-b04fcdf1a572"" sessionid=1629401503 proto="",  
"service" =\> """" service=""HTTPS"",  
"srcip" =\> "srcip=178.154.200.94",  
"srcport" =\> "srcport=41798",  
"policyid" =\> """" policyid=3"",  
"dstintf" =\> ""3 dstintf=""port2"",  
"dstport" =\> "7 dstport=44",  
"dstip" =\> """" dstip=45.249.108.19"",  
"time" =\> "time=23:26:26"  
}'''  
I am unable to understand even though I have defined columns and they are mapped correctly still why it is giving a weird output?

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 22, 2020, 12:05pm UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/2 "2020-05-22T12:05:42Z")

</div>

you might want to share a sample log that produces the output. also it helps a lot if you wrap the codes in \</\> for easier reading

---

<div class="post-metadata">

**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\
**Post date:** [May 26, 2020, 7:07am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/3 "2020-05-26T07:07:07Z")

</div>

\<date=2020-05-18,time=23:59:59,"logid=""0000000013""","type=""traffic""","subtype=""forward""","level=""notice""","vd=""root""",eventtime=1589826599,srcip=66.249.84.92,rcport=43417 s,"rcintf=""port1""","srcintfrole=""undefined""",dstip=45.249.108.197,dstport=443,"dstintf=""port2""","dstintfrole=""undefined"" poluuid=""9bd3b56e-98c4-51ea-b15c-b04fcdf1a572"" sessionid=1629563798 proto=6","action=""deny""",policyid=35,"policytype=""policy""","service=""HTTPS""","dstcountry=""India""","srccountry=""United States""","trandisp=""noop"" duration=0 sentbyte=0 rcvdbyte=0","sentpkt=0 appcat=""unscanned"" crscore=30 craction=131072 crlevel=""high"""\>

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 26, 2020, 9:48am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/4 "2020-05-26T09:48:40Z")

</div>

while it's csv, might be better to use [kv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-field_split) as your log is in key value pair

---

<div class="post-metadata">

**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\
**Post date:** [May 26, 2020, 10:28am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/5 "2020-05-26T10:28:06Z")

</div>

I tried kv filter but out of 33k documents its parsing only 12k documents and discarding the others

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 26, 2020, 12:11pm UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/6 "2020-05-26T12:11:50Z")

</div>

you could tag those failed with kv on failure and process with another filter [quote="Vikash\_Singh1, post:3, topic:233880, full:true"]  
\<date=2020-05-18,time=23:59:59,"logid=""0000000013""","type=""traffic""","subtype=""forward""","level=""notice""","vd=""root""",eventtime=1589826599,srcip=66.249.84.92,rcport=43417 s,"rcintf=""port1""","srcintfrole=""undefined""",dstip=45.249.108.197,dstport=443,"dstintf=""port2""","dstintfrole=""undefined"" poluuid=""9bd3b56e-98c4-51ea-b15c-b04fcdf1a572"" sessionid=1629563798 proto=6","action=""deny""",policyid=35,"policytype=""policy""","service=""HTTPS""","dstcountry=""India""","srccountry=""United States""","trandisp=""noop"" duration=0 sentbyte=0 rcvdbyte=0","sentpkt=0 appcat=""unscanned"" crscore=30 craction=131072 crlevel=""high"""\>  
[/quote]

if you’re using csv here, the fields will be have field\_name by column name and field\_value by csv value so you will end up with something like this

`“logid”: “ logid=""0000000013""`

---

<div class="post-metadata">

**Author:** ![Vikash\_Singh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vikash_singh1/32/42119_2.png) [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\
**Post date:** [May 27, 2020, 5:02am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/7 "2020-05-27T05:02:03Z")

</div>

> [@ptamba](#):
>
> field\_value

Even If i use "if" condition then which filter I should use? Can you help me?

Thanks in Advance

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 24, 2020, 5:02am UTC](https://discuss.elastic.co/t/problem-with-csv-filter/233880/8 "2020-06-24T05:02:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.