How to extract differnt log lines which are occuring between 2 lines of log files (Multiple grok with if condition)

[richagautam (Richa)]

Hi Team,

I am trying to parse multiple log files where log syntax are not same and extract different lines which is coming between 2 lines of log data. Below is the sample data

Feb 12 18:24:26 localhost.localdomain: SAMPLE START SampleTC()"
Feb 10 18:24:34 localhost.localdomain kibana[697]: WARNING in TestCase2()
Feb 10 18:24:37 localhost.localdomain kibana[697]: {"type":"log","@timestamp":"2018-02-08T12:39:37Z","tags":["warning","elasticsearch"],"pid":697,"message":"No living connections"}
Feb 10 18:24:40 localhost.localdomain kibana[697]: FAILED in SampleTC()
Feb 10 18:24:42 localhost.localdomain kibana[697]: {"type":"log","@timestamp":"2018-02-08T12:39:42Z","tags":["warning","elasticsearch"],"pid":697,"message":"Unable to revive connection: http://localhost:9200/"}
Feb 10 18:24:42 localhost.localdomain kibana[697]: {"type":"log","@timestamp":"2018-02-08T12:39:42Z","tags":["warning","elasticsearch"],"pid":697,"message":"No living connections"}
Feb 10 18:24:42 localhost.localdomain kibana[697]: FAILED String1 Strng2 in SampleTC()
Feb 12 18:26:46 localhost.localdomain: SAMPLE COMPLETED SampleTC()

I have to extract all the failure message ( like warning , fail, error) between SAMPLE START and SAMPLE COMPLETED lines which is coming the log file and maintain the order of data.

Currently I am able to extract relevant log lines by using multiple grow.
To achive the condition I am trying to implement below filter but I am not sure whether this approach is correct. Also I want to add TCNAME which I added in first grok in other grok, but I am not able to achieve this.

filter {
  grok {
    match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{GREEDYDATA:host} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: SAMPLE Start %{GREEDYDATA:DataString1}"}
    add_field => { "[TCNAME]" => "%{DataString1}" }
    remove_tag => ["_grokparsefailure"]
  }
  if [message] != "TESTCASE COMPLETED %{DataString}" {
    if "_grokparsefailure" in [tags]  {
      grok
       {
        match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{HOSTNAME:HOST} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: (?<DataString>WARNING %{GREEDYDATA})"}
        add_field => { "[TCNAME]" => "%{DataString1}" }
        remove_tag => ["_grokparsefailure"]
       }
    }
    if "_grokparsefailure" in [tags]  {
      grok {
        match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{HOSTNAME:HOST} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: (?<DataString>FAILED %{GREEDYDATA})"}
        add_field => { "[TCNAME]" => "%{DataString1}" }
        remove_tag => ["_grokparsefailure"]                 
      }
    }
  }
  if "_grokparsefailure" in [tags] {
    grok {
      match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{HOSTNAME:HOST} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: SAMPLE Completed %{GREEDYDATA:DataString}"}
      add_field => { "[TCNAME]" => "%{DataString}"}
      remove_tag => ["_grokparsefailure"]
    }
  }
  if "_grokparsefailure" in [tags] {
    drop {}
  }
  mutate{
    remove_field => ["@version", "host", "tags", "offset", "type", "input_type"]
  }
}

output {
  stdout { codec =>  rubydebug}
}

Thanks

[guyboertje]

Your initial assumptions need adjusting.

1. There is no guaranteed order to the processing of events in Logstash.
2. There is no state carried over from one event to the next.

That said, here is what I see...

• In the first grok, you have remove_tag => ["_grokparsefailure"] but this only takes effect on success so there is no tag to remove.
• if [message] != "TESTCASE COMPLETED %{DataString}" will always be true because the field DataString does not exist and so the resulting expression does not interpolate - the right hand side will always be the string literal "TESTCASE COMPLETED %{DataString}"
• the other conditional clauses may work but still not give the desired outcome.

You need to have the related lines as a set. To do this you will need a multiline function somewhere before the filter stage. You don't say what input you are using.

Once you have the related lines as one message string then I can advise further as you will need to remove the unwanted text from the inner sections to the message string. I have a filter pattern to do that.

However, lets first get the related lines joined as one - post the input you are using as the first step.