Multiline log

Elastic Stack Logstash
1

In a multiline log, can parsing or indexing of fields occur only in the first line? I want to parse fields present in the second or third line for example, what should I do?

2

That works but you may have to put (?m) at the beginning of the grok expression (which I assume is what you're using). For specific help please post an example message and your configuration.

3

Like this is my input{

file {
path => "/home/.*.com"
type => "framework"
codec => multiline {

pattern =>

"(?(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
%{MONTH} (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
%{HOUR}:%{MINUTE}:%{SECOND} %{YEAR})"
negate => true
what => "previous"
}
}

and this is my filter:
grok{
match => [ "message" ,
"(?(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
%{MONTH} (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
%{HOUR}:%{MINUTE}:%{SECOND} %{YEAR}) %{GREEDYDATA}
(ID=:)(?[0-9]{2,3}.[0-9]{2,3}.[0-9]{2,3}.[0-9]{2,3}([0-9]{1}))" ]
}

And suppose this is my log:

Thu Feb 25 15:00:03 2016 nbdnclmlmc;lm [INFO] kjnclkklmc
nckdnlmvl;
nkcjlkmcp;
nmdmc;c; RID=:10.88.81.211(0)
jhcbdkjnclkasdncl

4

Always always always post configurations (especially those with regular expressions) formatted as code! Your snippet above is not copy/pasteable and I had to rewrite it just to get it past the syntax check. Because I don't know exactly what you have I don't know exactly what was wrong with it, but the following works:

$ cat test.config 
input {
  stdin {
    codec => multiline {
      pattern => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
      negate => true
      what => "previous"
    }
  }
}
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => [
      "message",
      "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR} %{GREEDYDATA}; RID=:%{IP:ip}"
    ]
  }
}
$ cat data 
Thu Feb 25 15:00:03 2016 nbdnclmlmc;lm [INFO] kjnclkklmc
nckdnlmvl;
nkcjlkmcp;
nmdmc;c; RID=:10.88.81.211(0)
jhcbdkjnclkasdncl
Thu Feb 25 15:00:03 2016 nbdnclmlmc;lm [INFO] kjnclkklmc
nckdnlmvl;
nkcjlkmcp;
nmdmc;c; RID=:10.88.81.211(0)
jhcbdkjnclkasdncl
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Logstash startup completed
{
    "@timestamp" => "2016-03-04T06:37:33.464Z",
       "message" => "Thu Feb 25 15:00:03 2016 nbdnclmlmc;lm [INFO] kjnclkklmc\nnckdnlmvl;\nnkcjlkmcp;\nnmdmc;c; RID=:10.88.81.211(0)\njhcbdkjnclkasdncl",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "host" => "lnxolofon",
            "ip" => "10.88.81.211"
}
Logstash shutdown completed
5

GREEDYDATA does not capture the new line character, right? So how did it parse the RID?
You didn't even use (?m).

6

I would've expected (?m) to be necessary but I don't remember its exact semantics.