Grok pattern when last field can be multiline

Elastic Stack Logstash
1

I am trying to come up with a solution for a log file that is pipe delimited with the last field could be a multiline.

Example:
a|b|this is
multiline
c|d|no mutliline

Any pointers would be appreciated.

Also, since the file is fixed format (i.e. csv) is there any advantage of using grok pattern filter vs. a csv filter?

Thanks,
Frank.

2

How do you know that a line is a continuation of the previous line? That is doesn't contain a pipe character?

3

Here is what the last field looks like: Hope this helps.
Uncaught Exception in SilentScope: IESessionImpl:processDPCommandsEx: 0.368 Args=(Operation="306032") Vars=(Method Call Parameter Operation="306032")
EntityActivityNotifier.cpp:114:GenericContainer::EntityActivityNotifierWithTLSContext::~EntityActivityNotifierWithTLSContext(): TraceLog message 2796
|63ae678f-b640-74fc-d255-2954640f444e|2015 08 03 16:07:30:094|...... this line is a new event

4

So any line that does not begin with a pipe should be joined with the next line, then? That means you need a multiline filter or codec configured like this:

multiline {
  pattern => "^\|"
  negate => true
  what => "previous"
}
5

Thanks Magnus. The pipe delimiter on the second line in my example is actually is there b/c the first field on that line seems to be null. So, the newline does not begin with a pipe.

Here are two lines from the log. The first field is null. The second field is the guid followed by a timestamp, etc.

|f74d0e36-fd45-91fe-7cf3-fb0e2b6a7bdb|2015 08 03 16:07:30:094|Uncaught Exception in SilentScope: IESessionImpl:processDPCommandsEx: 0.368 Args=(Operation="306032") Vars=(Method Call Parameter Operation="306032")
EntityActivityNotifier.cpp:114:GenericContainer::EntityActivityNotifierWithTLSContext::~EntityActivityNotifierWithTLSContext(): TraceLog message 2796
|63ae678f-b640-74fc-d255-2954640f444e|2015 08 03 16:07:30:094|GenericContainer_EntityActivityNotifier:EntityActivityNotifierWithTLSContext_Destructor
EntityActivityNotifier.cpp:114:GenericContainer::EntityActivityNotifierWithTLSContext::~EntityActivityNotifierWithTLSContext(): TraceLog message 2797

Adding output from logstash. You can see how it broke up the message in two part. One thing I noticed is the \u001E at the end.

{"message":["|f74d0e36-fd45-91fe-7cf3-fb0e2b6a7bdb|2015 08 03 16:07:30:094|Uncaught Exception in SilentScope: IESessionImpl:processDPCommandsEx: 0.368 Args=(Operation=306032) Vars=(Method Call Parameter Operation=306032)\u001E\r"],"@version":"1","@timestamp":"2015-08-05T17:36:54.967Z","type":"webi","host":"mylaptop","path":"log_file","Location":null,"Guid":"f74d0e36-fd45-91fe-7cf3-fb0e2b6a7bdb","Time":"2015 08 03 16:07:30:094","Text":"Uncaught Exception in SilentScope: IESessionImpl:processDPCommandsEx: 0.368 Args=(Operation=306032) Vars=(Method Call Parameter Operation=306032)\u001E"}
{"message":["EntityActivityNotifier.cpp:114:GenericContainer::EntityActivityNotifierWithTLSContext::~EntityActivityNotifierWithTLSContext(): TraceLog message 2796\r"],"@version":"1","@timestamp":"2015-08-05T17:36:54.967Z","type":"webi","host":"mylaptop","path":"log_file","Location":"EntityActivityNotifier.cpp:114:GenericContainer::EntityActivityNotifierWithTLSContext::~EntityActivityNotifierWithTLSContext(): TraceLog message 2796"}

Update: I have tried the multiline and it does not seem to make one message.

Any other thoughts on this?

7

Update: I resolved the multiline issue. Followed what Magnum suggested. I had a typo .Duh!

Thanks again for helping with this.

Regards,
Frank.