If condition matching json with "in"

Elastic Stack Logstash
1

Hello all,

I have problem with a configuration which is supposed to select only lines containing a specific text inside. The source is a json_line file. What I want is that only the messages matching a given string are processed, like:

filter {

    if ("PUSH_BODY" in [message] ) {

        mutate { add_field => { "object_key" => "%{[@metadata][s3][key]}" }
                 add_field => { "topic" => "whatever" }
                 add_field => { "source_host" => "whatever.com" }
        }

                } else {
                drop{}
                }



}

so this should take only lines containing the string PUSH_BODY and drop everything else.

What could go wrong here? The input is a json formatted text file in custom format, which contains the following pattern:

"type":"PUSH_BODY"}

why this filter is not working?

thanks in advance,

FEM.

2

So the type field is "PUSH_BODY"? Then then conditional should look like this:

if [type] == "PUSH_BODY" {

Otherwise, please show what an event that doesn't match the conditional actually looks like. Use a stdout { codec => rubydebug } output or copy/paste from the JSON tab in Kibana's Discover panel.

3

Hi Magnus,

first, thank for the answer. :slight_smile:

The JSON looks like this:

{"id":"some-uuid","offset":"125018457","occurred":"2017-04-28T19:37:19.249Z","processed":"2017-04-28T19:37:20.362Z","body":{"payload":"somestring=","push_id":"someUUID","resource":"PUSH","trimmed":false},"type":"PUSH_BODY"}

I removed relevant data because is traffic from real users, sorry for that. I can confirm there is no other record containing "PUSH_BODY" . Basically I tried to use the message as a whole line, which should be possible even if the codec is json. For completeness, the input looks like:

input {
        s3 {
                bucket => "mybucket"
                codec => "json"
                region => "us-east-1"
                sincedb_path => "/somewhere/.sincedb_PUSHBODY"
                interval => 10
        }

}

regards,

FEM

4

Basically I tried to use the message as a whole line, which should be possible even if the codec is json.

If you've used the json or json_lines codec to deserialize a JSON string into discrete fields the original JSON string won't get preserved unless you make a copy of the field first.

5

Hi Magnus,

thanks again for the prompt answer... to my understanding: you mean the %{message} variable is emptied before of the filter , when the input has a codec of json?

Enrico

6

Yes. See for yourself.

If you have a discrete field with the exact value you're looking for you shouldn't be doing substring matching against the JSON string anyway.