Extract log between two patterns

Elastic Stack Logstash
1

Hi

We have logs that are very huge. I want to be able to extract logs [events] between two patterns.

For example:
2018/08/29 14:33:54.646 - some log
2018/08/29 14:33:58.809 - pattern1 -
rest of the log message under pattern 1
2018/08/29 14:30:58.338 - some log msg
2018/08/29 14:33:58.809 - pattern2 -
rest of the log message under pattern 2

I would like to extract -
2018/08/29 14:33:58.809 - pattern1 -
rest of the log message under pattern 1
2018/08/29 14:30:58.338 - some log msg
2018/08/29 14:33:58.809 - pattern2 -
rest of the log message under pattern 2

I know to drop the message if a pattern is matched but I do not know to extract all the events between two matching patterns. Can someone please help me with this? Thank you

2

You can have more than one patterns in the filter section so that only matching pattern will be parsed and rest of event will have _grokparsefailure that you can drop.

filter {
          grok {
		          match=>[
				            "message","%{pattern1}",
                            "message","%{pattern2}",
                            "message","%{pattern3}"
						 ]
		  }
		  if "_grokparsefailure" in [tags] {
                  drop { }
          }		

}
3

Thanks for replying.
With this, I can capture events matching pattern1 and pattern2 but I also need the events that fall between pattern1 and pattern2. There are no specific patterns for those events. So am unable to match those, am just able to extract events that match pattern1 and pattern2.

4

You can write these pattern1 pattern2 in if statements just like in _grokparsefailiure. And a general filter above. The ones that doesnt fall into ifs will fall into that general filter. Thats how "else" logic works in logstash I guess.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.