Doing mathematical ops using logstash filters

Elastic Stack Logstash
1

Hi,
I am facing a weired issue and need help of experts. I am trying to do numerical (INT) comparison in logstash filter file. Basically, I want to register only the haproxy events ( from its log file ) which take above 500ms of time_duration and only those calls which are less than 1000Bytes. I am using following in the logstash filter. But I see the result as given below. For now I am only using mutate, but ultimately I want to drop those events from the log before passing to elasticsearch.

====logstash filter config====
if 'time_duration' < '500' {
mutate { add_tag => [ "FAST" ] }
#drop { }
} else if 'time_duration' >= '500' {
mutate { add_tag => [ "SLOW" ] }
mutate { add_field => { "_ttl" => '365d' } }
}

For Main Haproxy

if [http_request] =~ /(?i)cmr.php/ {
drop { }
}
if 'bytes_read' > '1000' {
mutate { add_tag => [ "big" ] }
#drop { }
} else {
mutate { add_tag => [ "small" ] }
}
I am using haproxy config, which I don't think I need to print here. But for reference, the bytes_read and the time_duration are INT types.

The result that I get is this. Which clearly indicates that numerical comparison is not working. I tried putting the values in single ( ' ' ) quotes, in double quotes ( " " ). But nothing worked. I tried using the square brackets ( [ ] ) for the bytes_read and time_duration too. But nothing seems to work.
Here is the snippet of the stdout rubydebug log.

      "time_duration" => "11",
    "http_status_code" => "200",
          "bytes_read" => "219",

"captured_request_cookie" => "-",
"captured_response_cookie" => "-",
"termination_state" => "----",
"actconn" => "3349",
"feconn" => "1972",
"beconn" => "13",
"srvconn" => "1",
"retries" => "0",
"srv_queue" => "0",
"backend_queue" => "0",
"captured_request_headers" => "text/html",
"http_verb" => "GET",
"http_request" => "/campaign/rtb/adx/cmradc.php?vid=4613155667517436106",
"http_version" => "1.1",
"tags" => [
[0] "SUCCESS",
[1] "SLOW",
[2] "big"
],
"_ttl" => [
[0] "5d",
[1] "365d"
],
I would really appreciate if someone can send me some pointers to fix this issue.

2

if 'time_duration' < '500' {
mutate { add_tag => [ "FAST" ]
}

Two problems:

  • Fields are referred via the square bracket notation, i.e. [time_duration] in this case.
  • You need fields to be integers (or doubles) for numerical comparisons to work. See below.

I am using haproxy config, which I don't think I need to print here. But for reference, the bytes_read and the time_duration are INT types.

No, they're strings. That's obvious from the stdout output. The fact that a grok expression uses the predefined INT pattern merely means that the input string will be matched if it's an integer. The result will still be a string unless you add :int, i.e. %{INT:time_duration:int}. If you don't want to modify the original grok patterns you can use the mutate filter's convert option to convert string fields to integer or double fields.