HowTo: Integer comparison in logstash filter

I cannot get integer comparisons to work in logstash filter, using logstash 6.4.2.

input {
  beats {
    port => 5044
  }
}
filter {
    mutate {
        add_field => { "[@metadata][myLogLevel]" => 1 }

        add_field => { "[@metadata][day]" => "%{+dd}"}
        convert => { "[@metadata][day]" => "integer" }

        add_field => { "testDay" => "%{[@metadata][day]}" } # Copy to non-metadata field I can see in logs in kibana
        add_field => { "testDayFloat" => "%{[@metadata][day]}" }
        convert => { "testDay" => "integer" }
        convert => { "testDayFloat" => "float" }
    }

    if [@metadata][myLogLevel] < 10 { # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "whatever" }
        }
    }

    if [@metadata][day] < 10 { # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "0" }
        }
    }
    if [testDay] < 10 {  # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "0" }
        }
    }
    if [testDayFloat] < 10 { # this line fails with runtime error 'comparison of String with 10 failed'
        mutate {
            add_field => { "[@metadata][dayDiv10]" => "0" }
        }
    }
}
output {
    elasticsearch {
    ...
    }
}

I've tried the above, with each of the four failing lines included one at a time. As for as I understand from examples, questions and docs this should work, though I'm not sure about setting the metadata directly to an integer, the [@metadata][myLogLevel] line. But I've seen lots of examples doing the integer convert.

I get a runtime error when logstash reads a message, saying 'comparison of String with 10 failed'.

Any clue what I'm doing wrong?

It works for me when I do if-statement processing with ==, and =!, but not with >, presumably because both sides get converted to strings.

note the day is '06', not '6', I don't know if that gets converted to integer. After conversion to integer, kibana still shows a field containing '06'.

This is your issue. None of the converts are happening. A mutate filter does things in a fixed order, and once it has done all of that it decorates the event (that's the call to filter_matched()), which implements common options like add_field. That means the the convert executes before the add_field, so none of the fields exist when you try to convert them. Split your mutate into two

mutate {
    add_field => { "[@metadata][myLogLevel]" => 1 }
    add_field => { "[@metadata][day]" => "%{+dd}"}
    add_field => { "testDay" => "%{[@metadata][day]}" } # Copy to non-metadata field I can see in logs in kibana
    add_field => { "testDayFloat" => "%{[@metadata][day]}" }
}
mutate {
    convert => { "[@metadata][day]" => "integer" }
    convert => { "testDay" => "integer" }
    convert => { "testDayFloat" => "float" }
}

Awesome, thank you for the quick response and accurate solution. It works! Ahh so nice :slight_smile:

I have posted the question and answer here for better future reference for others (I kind of regret not posting it there in the first place):

If you wish you are welcome to answer it there with your profile, and I'll accept that, if you care about the public credits.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.