Csv file with multiline values getting parsed as single message

Elastic Stack Logstash
1

Hi again,

I am facing problem to parse a csv file that has values spread across multiline.
While trying with csv plugin ,the file is getting parsed as a single message(i.e: after the mentioned column names in csv plugin fields are getting parsed as column 15=>... ,column 16=>... ,column 17=>....,...... etc,

So my problem is how to write multiline pattern for csv to parse the field as different events!!!!????

Pls suggest something!!

2

Without knowing what the file looks like and what you expect Logstash to produce it's impossible to help.

3

Thanks for the reply,

I am sharing the configuration , log events and output i am getting:

csv Log events :

1654996090,3,22303-2306-22e6-8d34-12a456e152c0 ,100, AppRequest/Exception, 1, 10.1.1.30, FormattedMessage="Service SessionService: Raised Exception: Invalid token.
--- End of stack trace from previous location where exception was thrown ---
,AppRequest

2345160901,3,1w4ab1-1234-11e6-8d28-6e152c0, 20000, AppNotification/Notification, 2,10.1.1.56, FormattedMessage="When:Activation was called with parameters ReferenceException
--- End of stack trace from previous location where exception was thrown ---
,AppNotification

Configuration:

csv {

columns => ["time1","split","time2","id","name","level","ip","msg",task"]
separator => ","

}
...........

Output :

      "time1" => "1654996090",
       "split" => "3",
        "time2" => "22303-2306-22e6-8d34-12a456e152c0",
    "id" => "100",
  "name" => "AppRequest/ExceptionRaised",
       "level" => "1",
"ip" => "10.1.1.30",
     "msg" => "FormattedMessage=\\'Service SessionService: Raised Exception: Invalid token. --- End of stack trace from previou

s location where exception was thrown --- ",

   "task" => "AppRequest 2345160901",
    "column14" => "3",
    "column15" => "1w4ab1-1234-11e6-8d28-6e152c0",
    "column16" => "20000",
    "column17" => "AppNotification/NotificationFailure",
    "column18" => "2",
    "column19" => "10.1.1.56",
    "column20" => "FormattedMessage=\\'When:Activation was called with parameters ReferenceException --- End of stack trace from

previous location where exception was thrown --- ",

    "column21" => "AppNotification"

required output:

Here task field is taking next event minute column also(task" => "AppRequest 2345160901)
After AppRequest it should take as 2nd event, But it is expanding with column 14.....

So please suggest something so that it parse this multiline csv cinto different events!! :confused:

4

The configuration you posted is invalid (there's a double quote missing) and the log snippet is incomplete somehow because Logstash complains that it can't parse it.

$ cat data
1654996090,3,22303-2306-22e6-8d34-12a456e152c0 ,100, AppRequest/Exception, 1, 10.1.1.30, FormattedMessage=\"Service SessionService: Raised Exception: Invalid token.  --- End of stack trace from previous location where exception was thrown --- ,AppRequest
2345160901,3,1w4ab1-1234-11e6-8d28-6e152c0, 20000, AppNotification/Notification, 2,10.1.1.56, FormattedMessage=\"When:Activation was called with parameters ReferenceException --- End of stack trace from previous location where exception was thrown --- ,AppNotification
$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  csv {
    columns => ["time1","split","time2","id","name","level","ip","msg","task"]
    separator => ","
  }
}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
Error parsing csv {:field=>"message", :source=>"1654996090,3,22303-2306-22e6-8d34-12a456e152c0 ,100, AppRequest/Exception, 1, 10.1.1.30, FormattedMessage=\\\"Service SessionService: Raised Exception: Invalid token.  --- End of stack trace from previous location where exception was thrown --- ,AppRequest", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>, :level=>:warn}
Error parsing csv {:field=>"message", :source=>"2345160901,3,1w4ab1-1234-11e6-8d28-6e152c0, 20000, AppNotification/Notification, 2,10.1.1.56, FormattedMessage=\\\"When:Activation was called with parameters ReferenceException --- End of stack trace from previous location where exception was thrown --- ,AppNotification", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>, :level=>:warn}
{
       "message" => "1654996090,3,22303-2306-22e6-8d34-12a456e152c0 ,100, AppRequest/Exception, 1, 10.1.1.30, FormattedMessage=\\\"Service SessionService: Raised Exception: Invalid token.  --- End of stack trace from previous location where exception was thrown --- ,AppRequest",
      "@version" => "1",
    "@timestamp" => "2016-09-22T07:12:45.601Z",
          "host" => "lnxolofon",
          "tags" => [
        [0] "_csvparsefailure"
    ]
}
{
       "message" => "2345160901,3,1w4ab1-1234-11e6-8d28-6e152c0, 20000, AppNotification/Notification, 2,10.1.1.56, FormattedMessage=\\\"When:Activation was called with parameters ReferenceException --- End of stack trace from previous location where exception was thrown --- ,AppNotification",
      "@version" => "1",
    "@timestamp" => "2016-09-22T07:12:45.639Z",
          "host" => "lnxolofon",
          "tags" => [
        [0] "_csvparsefailure"
    ]
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

Please edit your post so that it reflects reality and so that someone else can reproduce the problem.

5

Ok sorry there was a missing double quote in task column so i have added it :

csv
{
columns => ["time1","split","time2","id","name","level","ip","msg","task"] // i changed from task" to "task"
separator => ","
}

Following is snippet for input multine codec :
input
{
codec => multiline
{

pattern =>'^("time1")("split")("time2")("id")("name")("level")("ip")("msg")("task")'
negate=>true
what => "next"
}

}

Output:

     "time1" => "1654996090",
     "split" => "3",
     "time2" => "22303-2306-22e6-8d34-12a456e152c0 ",
        "id" => "100",
      "name" => " AppRequest/Exception",
     "level" => " 1",
        "ip" => " 10.1.1.30",
       "msg" => " FormattedMessage=\\'Service SessionService: Raised Exception: Invalid token. --- End of stack trace from previous lo

cation where exception was thrown --- ",
"task" => "AppRequest 2345160901",
"column10" => "3",
"column11" => "1w4ab1-1234-11e6-8d28-6e152c0",
"column12" => " 20000",
"column13" => " AppNotification/Notification",
"column14" => " 2",
"column15" => "10.1.1.56",
"column16" => " FormattedMessage=\'When:Activation was called with parameters ReferenceException --- End of stack trace from previo
us location where exception was thrown ---"

Thanks again & please suggest why multiline is not working!!!