Grok with Else conditionals. How?

[mgagliardi (Matt Gagliardi)]

Hello all. I'm new to Logstash/ELK and a pretty weak coder. I'm trying to tidy up a filter which uses Grok and running into an error. Basically I've got a given log file and a filter that works but after adding a couple of addidtional Grok statements (to mark two specific events for use with Elapsed by adding tags to them) it ends up marking about 98% of events as "_grokparsefailed". I know why...because 98% of events don't contain info such that they should be marked/get tags added.

Anyway...I thought the way to clean this up would be to move all of my Grok statements into a single block featuring "else" conditionals. Essentially the flow looks like "Mark for event 1? -> else -> Mark for even 2? -> else -> Final Grok statement -> out to rest of filter". It looks to me like it should work. But when I run "logstash configtest" I keep getting an error which indicates it's expecting something "Expected one of #, => at line X, column Y (byte Z) after filter {". Which brings me back to that whole I'm not a very capable coder bit.

If anyone could take a fast look at the paste below and point out what I'm doing wrong I'd really appreciate it. FWIW I have searched and read the docs...I've been stuck on this for about 4 hours. TIA!

filter {
  if [type] == "7x_SPA" {
    grok {
	  match => { "message" => ["^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>\bViPER:\sApp\sfirst\simage\savailable\b$"]}
	  add_tag => [ "firstImage" ]
	     }
	else {
      grok {
	    match => { "message" => ["^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>\bViPER:\sApp\slaunch\sinitial\srequest\b$"]}
	    add_tag => [ "launchReq" ]
	       }
	     }
	else {
      grok {
        break_on_match => true
        match => { "message" => ["^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>[^\;]*$)",
                               "^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>%{GREEDYDATA});\s(?<spaJson>%{GREEDYDATA})"
                                ]
                 }
           }
	     }
	date {
      match => [ 'logTime', 'yyyy-MM-dd HH:mm:ss.SSSSSSZ' ]
      remove_field => [ 'logTime' ]
    }
	elapsed {
	  start_tag => "launchReq"
	  end_tag => "firstImage"
	  unique_id_field => "logSheet"
	}
	json {
	  source => "spaJson"
	  target => "parsedJson"
    }
  }
}

[magnusbaeck (Magnus Bäck)]

else blocks are connected to if conditionals, of which you have none. Try this pattern instead:

grok {
  ...
}
if "_grokparsefailure" in [tags] {
  grok {
    ...
  }
}
...

In other words, try new grok filters until the event gets a _grokparsefailure.

[mgagliardi (Matt Gagliardi)]

Thank you for the quick reply Magnus, I think I see what you're suggesting and I'm about to go try it. May I ask though, why would I want a _grokparsefailure?

[magnusbaeck (Magnus Bäck)]

You don't want the _grokparsefailure tag, but that's grok's way of signalling that none of the expressions matched.

But wait. The logic in my example is flawed and won't work if you have three or more grok filters. Do this instead:

grok {
  ...
  tag_on_failure => []
  add_tag => ["grok_success"]
}
if "grok_success" not in [tags] {
  grok {
    ...
    tag_on_failure => []
    add_tag => ["grok_success"]
  }
}
...

So, continue trying new grok filters until the event gets a grok_success tag, indicating that grok succeeded.

[mgagliardi (Matt Gagliardi)]

Thanks again Magnus, the suggestions are very much appreciated.

[mgagliardi (Matt Gagliardi)]

One last snag, if I get through this it's done. The first two Grok statements are now working properly, adding the tags I want added (for use with Elapsed). The problem is that they're also being sent through the final Grok statement and ending up with duplicate information in certain tags. For example, I have a tag named "slotID", in the desired state that has a value but only has it once. The events that do not match the first two Grok statements (and thus fall to the third for processing) are in that state. But all of the events that match one of the first two Grok statements have multiple entries in slotID (and other tags as well).

I'm trying to keep those events from going through the third Grok statment with this:

  if "firstImage" or "launchReq" not in [tags] {
  grok {
    break_on_match => true
    match => { "message" => ["^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>[^\;]*$)",
                             "^(?<logTime>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:%{MINUTE}:%{SECOND}%{ISO8601_TIMEZONE})\t(?<logSheet>%{BASE10NUM})\t(?<slotID>%{NOTSPACE})\t(?<category>%{WORD})\t(?<Message>%{GREEDYDATA});\s(?<spaJson>%{GREEDYDATA})"
								]
             }
      }
	     }

But based on my results that doens't seem to be working. Any ideas?

[magnusbaeck (Magnus Bäck)]

if "firstImage" or "launchReq" not in [tags] {

Needs to be:

if "firstImage" not in [tags] and "launchReq" not in [tags] {

[mgagliardi (Matt Gagliardi)]

FWIW Magnus, the events in question should only have one of those tags...either firstImage or launchReq. Still use the and rather than the or?

EDIT:
Well damn, there it is. Don't question the master Thanks Magnus, that's got it. I owe you.