How to filter the logs which has the numeric value greater than specific number in particular field?

Elastic Stack Logstash
1

I am running tomcat service in client. I would like to monitor tomcat access logs and filter the logs which has response time greater than specific number.

e.g : I have the tomcat access log.

192.168.1.10 - - [26/May/2015:21:56:51 -0700] "POST /url HTTP/1.1" 200 50

In the example, it has the response time 50 in the 10th field. i would like to check condition at 10th filed. if the value exceeds the limit(specific number), i need to filter that logs.

Let me, what is the filter(grok or range) will be useful for that and also let me know the syntax.

Thanks in advance.

2

Hello,

That was quite easy, and grok filters for tomcat/apache logs are easy to find on google.

Here is full example of what you did, with mathing events with time above 10. I had used 'mutate' and 'add_tag' to show that.

The rules:


The documentation and examples :

3

Hello,
It matches all logs which means it sends all the logs to elasticsearch server and adds tag "matched_above_10" if the response_time greater than 10.

Instead of that, it will send response_time greater than 10 logs only to elasticsearch server(i.e It would drop other logs) and will add tag "matched_above_10".

4

I used simple regex to grab greater than 50 response value logs and removed "_grokparsefailure" tags. Now i got the expected results.

filter {
if [type] == "tomcat_response" {
grok {
match => { "message" => "%{IP} - - [%{HTTPDATE}] "%{DATA}" %{NUMBER} ([6-9]\d|\d{3,})" }
add_tag => "slowresponse"
}
if "_grokparsefailure" in [tags] {
drop { }
}
}
}

Thanks for your assist.

5

I used simple regex to grab greater than 50 response value logs and removed "_grokparsefailure" tags. Now i got the expected results.

Yes, but that's not a very nice way of doing it. Given a proper grok expression that extracts the fields in the log (which you should have anyway) you can just do this:

filter {
  if [type] == "tomcat_response" and [response_time] > 50 {
    mutate {
      add_tag => "slowresponse"
    }
  } else {
    drop { }
  }
}
6

I had idea to try else condition today. But you got it. The new syntax makes sense.
It is perfect.

Thanks.