General question on filter order (with grok filters)

Elastic Stack Logstash
1

Hi,
I am new to the ELK stack and loving and learning it. I have a likely basic question, so if you know of a good thread that I missed I would love if you could let me know of it.

The question is how are filter processed.

filter {
   if "kibana" in [message] and [log_level] == "-" {
    drop {}
   }

#I assume anything that fits the above statement is just gone

  grok {
    #Nov 14 20:29:34 mydomain smartd[12031]: Device: /dev/sda [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 83 to 100
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:server} %{DATA:process}\[%{INT:processpid:int}\]: Device: %{DATA:harddisk} \[%{DATA:interface}\], %{GREEDYDATA:smartattrmsg}: %{INT:smartprefailattrnr:int} %{DATA$
  }

#anything that fits the above filter now has on top of the message field additional fields as defined by the filter until output is finished. Messages that do not fit the filter continue.

#do I need to test if a filed exists, without the system messages that didn't fit the previous filter throwing an error? So in the below case just writing "if [smartprefailattrnr] = 1". That should be faster then again running through the full message?

  if [message] =~ "Raw_Read_Error_Rate" {
    if [oldrate] > 60 or [newrate] > 60{
      mutate{
        replace => { "log_level" => "INFO" }
      }
    }
    #no else, because that is already an ERROR from the standard grok filter
  }
 }

I could then add another and another and another grok filter here, right? But it must all be at the same level as the initial grok filter?

Is there a best practice how to work with grok filters. I was wondering if maybe once I run the first grok filter, I should overwrite the message with smartattrmsg, to avoid that the system message triggers another following grok filter that is again looking at the full message.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.