HowTo: Integer comparison in logstash filter

Elastic Stack Logstash
1

I cannot get integer comparisons to work in logstash filter, using logstash 6.4.2.

input {
  beats {
    port => 5044
  }
}
filter {
    mutate {
        add_field => { "[@metadata][myLogLevel]" => 1 }

        add_field => { "[@metadata][day]" => "%{+dd}"}
        convert => { "[@metadata][day]" => "integer" }

        add_field => { "testDay" => "%{[@metadata][day]}" } # Copy to non-metadata field I can see in logs in kibana
        add_field => { "testDayFloat" => "%{[@metadata][day]}" }
        convert => { "testDay" => "integer" }
        convert => { "testDayFloat" => "float" }
    }

    if [@metadata][myLogLevel] < 10 { # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "whatever" }
        }
    }

    if [@metadata][day] < 10 { # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "0" }
        }
    }
    if [testDay] < 10 {  # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "0" }
        }
    }
    if [testDayFloat] < 10 { # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "0" }
        }
    }
}
output {
    elasticsearch {
    ...
    }
}

I've tried the above, with each of the four failing lines included one at a time. As for as I understand from examples, questions and docs this should work, though I'm not sure about setting the metadata directly to an integer, the [@metadata][myLogLevel] line. But I've seen lots of examples doing the integer convert.

I get a runtime error when logstash reads a message, saying 'comparison of String with 10 failed'.

Any clue what I'm doing wrong?

It works for me when I do if-statement processing with ==, and =!, but not with >, presumably because both sides get converted to strings.

note the day is '06', not '6', I don't know if that gets converted to integer. After conversion to integer, kibana still shows a field containing '06'.

2

This is your issue. None of the converts are happening. A mutate filter does things in a fixed order, and once it has done all of that it decorates the event (that's the call to filter_matched()), which implements common options like add_field. That means the the convert executes before the add_field, so none of the fields exist when you try to convert them. Split your mutate into two

mutate {
    add_field => { "[@metadata][myLogLevel]" => 1 }
    add_field => { "[@metadata][day]" => "%{+dd}"}
    add_field => { "testDay" => "%{[@metadata][day]}" } # Copy to non-metadata field I can see in logs in kibana
    add_field => { "testDayFloat" => "%{[@metadata][day]}" }
}
mutate {
    convert => { "[@metadata][day]" => "integer" }
    convert => { "testDay" => "integer" }
    convert => { "testDayFloat" => "float" }
}
3

Awesome, thank you for the quick response and accurate solution. It works! Ahh so nice :slight_smile:

I have posted the question and answer here for better future reference for others (I kind of regret not posting it there in the first place):

If you wish you are welcome to answer it there with your profile, and I'll accept that, if you care about the public credits.