Parsing a multiline log

Elastic Stack Logstash
1

How can I parse some information present in the last line of a multiline message. For example, the log format is:
the first line contains the timestamp, the log level and some other important information, which is followed by a multiline stack-trace, and within that stack trace is a specific pattern. I want to extract that pattern. How can I do it?

I have used the multiline codec, but it just combines the log to one message, but the parsing of patterns happen only in the first line of the log. Can someone please help me?

2

You may have to put (?m) at the beginning of the pattern. If you show an example input line and your current configuration it'll be less work to figure out what's wrong and it's more likely that someone will help you.

3

For example:
log:
12 feb 2016 ......(something)
(line 1)
(line 2)
(line 3)
line 4 ........... ID: 123abd(a long string with no spaces)
(line 5)

Now after applying the multiline codec, if my pattern is "%{Timestamp} %{GREEDYDATA} (?ID:[\S]+)"
it fails.
How do i do it?

4

Please provide your current configuration as requested earlier.

(?ID:[\S]+)"

According to this pattern there can't be any spaces after "ID:" but in your example line there is a space.

5

If this is the pattern, then?
It still shows "_grokparsefailure" !!!

6

For the third time, please provide your current configuration.

7

The configuration file?

Like this is my input{

file {
path => "/home/.*.com"
type => "framework"
codec => multiline {
pattern => "(?(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?) %{MONTH} (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) %{HOUR}:%{MINUTE}:%{SECOND} %{YEAR})"
negate => true
what => "previous"
}
}

and this is my filter:
grok{
match => [ "message" , "(?(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?) %{MONTH} (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) %{HOUR}:%{MINUTE}:%{SECOND} %{YEAR}) %{GREEDYDATA} (ID=:)(?[0-9]{2,3}.[0-9]{2,3}.[0-9]{2,3}.[0-9]{2,3}([0-9]{1}))" ]
}

The timestamp format is okay. Like it is parsed according to the format I have. My question is how to I fetch the "ID" present in any line other than the first line?

8

The configuration file?

Yes!

The configuration you just posted doesn't match the example your posted earlier and I'm all out of patience. Good luck.

9

This is an actual log:
Thu Feb 25 15:00:03 2016 nbdnclmlmc;lm [INFO] kjnclkklmc
nckdnlmvl;
nkcjlkmcp;
nmdmc;c; RID=:10.88.81.211(0)
jhcbdkjnclkasdncl

Okay, for the last time. Please help me out.

10

Please help me out.

11

It's actually a very good question. I am stuck on it too. What's the way out?