If condition matching json with "in"

Hello all,

I have problem with a configuration which is supposed to select only lines containing a specific text inside. The source is a json_line file. What I want is that only the messages matching a given string are processed, like:

filter {

    if ("PUSH_BODY" in [message] ) {

        mutate { add_field => { "object_key" => "%{[@metadata][s3][key]}" }
                 add_field => { "topic" => "whatever" }
                 add_field => { "source_host" => "whatever.com" }
        }

                } else {
                drop{}
                }



}

so this should take only lines containing the string PUSH_BODY and drop everything else.

What could go wrong here? The input is a json formatted text file in custom format, which contains the following pattern:

"type":"PUSH_BODY"}

why this filter is not working?

thanks in advance,

FEM.

So the type field is "PUSH_BODY"? Then then conditional should look like this:

if [type] == "PUSH_BODY" {

Otherwise, please show what an event that doesn't match the conditional actually looks like. Use a stdout { codec => rubydebug } output or copy/paste from the JSON tab in Kibana's Discover panel.

Hi Magnus,

first, thank for the answer. :slight_smile:

The JSON looks like this:

{"id":"some-uuid","offset":"125018457","occurred":"2017-04-28T19:37:19.249Z","processed":"2017-04-28T19:37:20.362Z","body":{"payload":"somestring=","push_id":"someUUID","resource":"PUSH","trimmed":false},"type":"PUSH_BODY"}

I removed relevant data because is traffic from real users, sorry for that. I can confirm there is no other record containing "PUSH_BODY" . Basically I tried to use the message as a whole line, which should be possible even if the codec is json. For completeness, the input looks like:

input {
        s3 {
                bucket => "mybucket"
                codec => "json"
                region => "us-east-1"
                sincedb_path => "/somewhere/.sincedb_PUSHBODY"
                interval => 10
        }

}

regards,

FEM

Basically I tried to use the message as a whole line, which should be possible even if the codec is json.

If you've used the json or json_lines codec to deserialize a JSON string into discrete fields the original JSON string won't get preserved unless you make a copy of the field first.

Hi Magnus,

thanks again for the prompt answer... to my understanding: you mean the %{message} variable is emptied before of the filter , when the input has a codec of json?

Enrico

Yes. See for yourself.

If you have a discrete field with the exact value you're looking for you shouldn't be doing substring matching against the JSON string anyway.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.