Matching two different patterns in one config

Elastic Stack Logstash
1

I have been trying several things to no avail - need some help.

My log file contains both successful and unsuccessful transactions - trouble is that I have to use a different match pattern to recognize Success and Failure. Thus I have defined two separate Match statements - top one for Successful transaction and the bottom one for Failed transaction. I was trying to condition things to produce the fields I need with values that make sense.

I would like to get the following fields out of each one of the matches:
tslice
senderid
accountnbr
status (success/failure)

How can I set "success" or "failure" in the status field based on match results? I managed to get 'statusword field' which contains some words in it that I could use.... was trying some conditionals, but it doesn't do anything. There must be a better way.

Here is my current Filter config:

filter {
  grok {
    match => [
      "message",
      "(?<tslice>%{DATE} %{TIME}) %{GREEDYDATA}(?<senderid>_SLIC-%{WORD}-BILLING-\d{5}:)%{GREEDYDATA}, account= %{NUMBER:accountnbr} %{GREEDYDATA} (%{WORD:statusword})$"
    ]
    match => [
          "message",
          "(?<tslice>%{DATE} %{TIME}) %{GREEDYDATA}(?<senderid>_SLIC-%{WORD}-BILLING-\d{5}:)%{GREEDYDATA} missing data in BMSIW %{GREEDYDATA} account id=%{NUMBER:accountnbr} %{GREEDYDATA} (?<statusword>skipping this account)"
    ]
  }
 
    if ("statusword" =~ "skipping") {
      mutate {
        add_field => { "status" => "failure" }
      }
    } else if ("statusword" == "OK") {
      mutate {
        add_field => { "status" => "success" }
      }
    } else {
        drop {}
    }
      #if "_grokparsefailure" in [tags] {
      #   drop { }
      #}
}
2

Maybe try this:

match => [
      "message", "(?<tslice>%{DATE} %{TIME}) %{GREEDYDATA}(?<senderid>_SLIC-%{WORD}-BILLING-\d{5}:)%{GREEDYDATA}, account= %{NUMBER:accountnbr} %{GREEDYDATA} (%{WORD:statusword})$",
      "message", "(?<tslice>%{DATE} %{TIME}) %{GREEDYDATA}(?<senderid>_SLIC-%{WORD}-BILLING-\d{5}:)%{GREEDYDATA} missing data in BMSIW %{GREEDYDATA} account id=%{NUMBER:accountnbr} %{GREEDYDATA} (?<statusword>skipping this account)"
    ]

and then instead of wrapping with "" use "["

if [statusword] =~ "skipping" {
      mutate {
        add_field => { "status" => "failure" }
      }
    } else if ([statusword]  == "OK") {
      mutate {
        add_field => { "status" => "success" }
      }
    } else {
        drop {}
    }

Thats how I have been doing it

3

Those two statements under 'match' separated by comma didn't work for me....

I got this to work (partially) - the problem I still have is that the output contains success/failures PLUS all those messages that didn't match one pattern or the other pattern.

I cannot exclude anything by checking for _grokparsefailure, because that's present for every single line now that I am looking for two things... :frowning:

Here is my filter config - it gets me what I want and what I don't want, and then I'm sending everything to ES: good, bad and everything in between.... how can I restrict output to just status = success/failure?! I tried conditioning the Output, but it wouldn't write anything out.... maybe my syntax was wrong...

filter {
  grok {
    # Look for success
    match => [
      "message", "(?<tslice>%{DATE} %{TIME}) %{GREEDYDATA}(?<senderid>_SLIC-%{WORD}-BILLING-\d{5}:)%{GREEDYDATA}, account= %{NUMBER:accountnbr} %{GREEDYDATA} (%{WORD:statusword})$"
    ]
  }
 
if ([statusword] == "OK") {
  mutate {
    add_field => { "status" => "success" }
    # Wish there was a way to Exit Out at this point so it would skip the next filter alltogether
  }
} 
    
  grok {
    # Look for failure
    match => [
      "message", "(?<tslice>%{DATE} %{TIME}) %{GREEDYDATA}(?<senderid>_SLIC-%{WORD}-BILLING-\d{5}:)%{GREEDYDATA} missing data in BMSIW %{GREEDYDATA} account id=%{NUMBER:accountnbr} %{GREEDYDATA} (?<statusword>skipping this account)"
    ]
  }    
    
if ([statusword] =~ "skipping") {
  mutate {
    add_field => { "status" => "failure" }
  }
}  

}

    
output { 
    #if [status] == "success" or [status] == "failure" {
    #    elasticsearch {
    #        host => localhost
    #        index => "billing08"
    #    }
    #}
        elasticsearch {
            host => localhost
            index => "billing08"
        }
    stdout { 
        codec => rubydebug 
    } 

}
4

Answering my own question here, I've placed conditions on the Output, so that output is directed to ES only under the desired conditions - it works fine.

output { 
    if ("success" in [status]) or ("failure" in [status]) {
        elasticsearch {
            host => localhost
            index => "billing08"
        }
    }
    stdout { 
        codec => rubydebug 
    } 
}

Related Question: I have two filter matches defined in my config. Is there a way to skip over the second filter if I discover the right conditions in the first filter?! I don't think I want to drop {} to event, I just don't need the rest of the filtering criteria to be evaluated once I find the match in the first filter. Is there way as skip the rest of filtering and go directly to the Output? See the config above in message nbr 3 - I've placed a comment there: "# Wish there was a way to Exit Out at this point..."

5

James,

Sorry for late reply busy with new school year. There is a config option in grok that does break on match I believe that is your use case. But when I look at your config you are trying grok twice with the same message. Can you paste some error messages you get when you combine them as I demonstrated earlier. It will solve the break on match problem you face.