New to ELK - Basic issue with parsing csv file in Logstash

Elastic Stack Logstash
1

Need help in decoding the error message thrown by Logstash.

// single record in the csv file

1-9,1,8,029030000000201567295889,,,,,,,,,,,,,,,,,,,,,,,,,29,1|1|17742639738,sip:+17742639738@vzims.com,,1|1|19782010063,tel:+19782010063,,20190831235809046+0000,20190831235810483+0000,0,O-RMS,,,29-136-20-3-1048624,,,,,,,,*;+g.3gpp.icsi-ref="urn%3Aurn-7%3A3gpp-service.ims.icsi.oma.cpm.msg";explicit;require,IN,,34ce08c7-ac26-4def-a92a-596fad4547e3,,,,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,64e38dfe-92df-4d5a-836e-dbf2808a5a99,,,,,311480444315571@vzims.com,,0,0,No,,,0,0|0|[2001,MESSAGE,20190831235809044+0000,0,tel:+17742639738,0,73,GtMl_pQGc--xnPYNHG7_lg..@2600:1000:102e:1ba::1c4:e62a,message/cpim,1,0|0|<isc@njwthpscscf1640.vzims.com,MESSAGE,20190831235809048+0000,200,tel:+17742639738,0,73,GtMl_pQGc--xnPYNHG7_lg..@2600:1000:102e:1ba::1c4:e62a~rmso(29-136-20-3-0-3-1),message/cpim,,,1,,,,,,,,,,,,,,,,0,,,,,,,20190831235810481+0000,20190831235809044+0000,20190831235810483+0000,0,,2,0,0,,20190831235809045+0000,CPM-serv/OMA1.0,de3e8c48-6484-4bd5-916b-e926ad9a5578,1,0,1,0,ccf="aaa://ccf1.NEE29.vzims.com";ccf="aaa://ccf2.NEE29.vzims.com";ecf="aaa://ecf1.NEE29.vzims.com";ecf="aaa://ecf2.NEE29.vzi",,IEEE-802.11,,,,,,0,0,0,0,0,0,0,,,,,,,,,,,PageMode,IEEE-802.11;i-wlan-node-,,,,,,,0,,0,,213-233-227-217-339-219-342-338-342-253-350-

//Error seen on Logstash
[2019-09-06T12:12:47,018][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"1-9,1,8,029030000000201567295889,,,,,,,,,,,,,,,,,,,,,,,,,29,1|1|17742639738,sip:+17742639738@vzims.com,,1|1|19782010063,tel:+19782010063,,20190831235809046+0000,20190831235810483+0000,0,O-RMS,,,29-136-20-3-1048624,,,,,,,,*;+g.3gpp.icsi-ref=\"urn%3Aurn-7%3A3gpp-service.ims.icsi.oma.cpm.msg\";explicit;require,IN,,34ce08c7-ac26-4def-a92a-596fad4547e3,,,,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,64e38dfe-92df-4d5a-836e-dbf2808a5a99,,,,,311480444315571@vzims.com,,0,0,No,,,0,0|0|[2001,MESSAGE,20190831235809044+0000,0,tel:+17742639738,0,73,GtMl_pQGc--xnPYNHG7_lg..@2600:1000:102e:1ba::1c4:e62a,message/cpim,1,0|0|<isc@njwthpscscf1640.vzims.com,MESSAGE,20190831235809048+0000,200,tel:+17742639738,0,73,GtMl_pQGc--xnPYNHG7_lg..@2600:1000:102e:1ba::1c4:e62a~rmso(29-136-20-3-0-3-1),message/cpim,,,1,,,,,,,,,,,,,,,,0,,,,,,,20190831235810481+0000,20190831235809044+0000,20190831235810483+0000,0,,2,0,0,,20190831235809045+0000,CPM-serv/OMA1.0,de3e8c48-6484-4bd5-916b-e926ad9a5578,1,0,1,0,ccf=\"aaa://ccf1.NEE29.vzims.com\";ccf=\"aaa://ccf2.NEE29.vzims.com\";ecf=\"aaa://ecf1.NEE29.vzims.com\";ecf=\"aaa://ecf2.NEE29.vzi\",,IEEE-802.11,,,,,,0,0,0,0,0,0,0,,,,,,,,,,,PageMode,IEEE-802.11;i-wlan-node-,,,,,,,0,,0,,213-233-227-217-339-219-342-338-342-253-350-\r", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}

2

The default quote_char for a csv filter is double quote. You cannot quote part of a field. You have to quote the entire field and escape double quotes with it.

However, from the look of it, you do not actually need quoting. So you could define quote_char as some other character that does not occur in your input (^ maybe?), or else you could use mutate+gsub to replace the double quotes with something else

mutate { gsub => [ "message", '"', "!!doubleQuotes!!" ] }

but then you need to iterate over the fields afterwards and reverse the gsub.

1 Like
3

Thanks a lot for the help Badger. I can do without the double quotes and I have removed them.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.