How many matches, we can have in grok filter - as example below


(R!) #1

I have assorted logs coming to syslog file as one all.log, i am looking sort using different grok match, is this possible as below : match ?

input {
file {
path => "/var/syslog-ng/raw/*.log"
start_position => "beginning"
type => "logstash-syslog"
tags => [ "logstash-syslog" ]
}
}
filter {
grok {
match => {

                   [ "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:device_src} %{NUMBER} %{SYSLOGTIMESTAMP} %{WORD}: %{GREEDYDATA:syslog_message}",
                    "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:device_src} %{GREEDYDATA:syslog_message}"]
            }
    }
    date {
             match => ["syslog_timestamp", "MMM dd HH:mm:ss" ]
    }
    ruby {
  code => "event.remove('type')"

}
}

output {
elasticsearch {
hosts => ["192.168.1.75:9200"]
index => "logstash-syslog"
}
stdout {
codec => rubydebug
}
}


Grok : Extract multiple lines format occurring between two lines of log data
(Magnus Bäck) #2

The documentation of the match option in the grok filter documentation contains an example of how to specify more than one expression.


(R!) #3

this what i see in the document.

grok {
match => {
"message" => ["expr1", "expr2", ..., "exprN"]
}
}

is that mean, i can have 100 expression ? 30 different logs and log format stored in the same file

Curiosity do i consider delay, processing 100 expressions ?


(Magnus Bäck) #4

is that mean, i can have 100 expression ? 30 different logs and log format stored in the same file

Yes.

Curiosity do i consider delay, processing 100 expressions ?

Of course, but the grok filter doesn't always test all expressions. It terminates the search as soon as there's a match, so you'll want to sort the expressions in "most likely to match" order.


(R!) #5

Appreciate your reply.

"most likely to match" order

This means, we should not have any generic expression which consumes more time to recursive lookup.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.