Dear all,
I have configured Logstash for a csv input file. Each field is double quoted, comma separated and terminated by /r/n. Sometimes the last field is a multiline text.
This is an example record (timestamp is on the same line, but after posting it has been truncated ):
"22-07-2016","07:27:41","6612","X5X","DEBUG","NO","LEp2xY","86","STATUSCHANGED","241108300829090309030000110000011010725110875087609140914000011000001101
072611078907880884088300101q000p01101 0000p10p000000000000100001110000011011
,mdsdf"
This is my file configuration in logstash.conf:
file {
path => "/logs/debug.log"
type => "debug"
start_position => "beginning"
codec => multiline {
pattern => "^\x22[0-9]{1,2}-[0-9]{1,2}-[0-9]{4}.*"
negate => true
what => previous
}
}
The configuration works fine for single line records (Kibana shown data parsed), but multiline records generate an error:
{:timestamp=>"2016-07-22T09:27:42.936000+0200", :message=>"Error parsing csv", :field=>"message", :source=>"\"22-07-2016\",\"07:27:41\",\"6612\",\"X5X\",\"DEBUG\",\"NO\",\"LEp2xY\",\"86\",\"STATUSCHANGED\",\"24110830082909030903000011000001101\r\n0725110875087609140914000011000001101\r\n072611078907880884088300101q000p01101\r\n00000000000000001010000000130 q0q1101\r\b0000p10p00000000000010000111000001101\r\n,mdsdf\"\r", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>, :level=>:warn}
As far as I understand, seems that the last double quote is not aggregated to the previous line, as stated in config file.
Could you please help me solving this issue ?
Thank you in advance.
FG