If Condition in JSON filter

Elastic Stack Logstash
1

Hello,

I am ingesting JSON data to logstash, and I am using JSON filter.
In the JSON data, when the KEY is either Value 1 or Value 2, I should add a field, and if this key is missing in the logs, I will have to drop it. Please advise on how to code this.

Below is code, and its not effective.

 if "true" in ["adf"] {
                mutate {
                        add_field => {"TrueCondition" => "Test True" }
                }
        }
         if "false" in ["adf"] {
                mutate {
                        add_field => {"FalseCondition" => "Test False" }
                }
        }

Data captured in the JSON event: "adf": false or "adf": true
Thanks in Advance.

--
Siddarth

2

in has two uses, one is an array membership test, the other is testing whether a substring exists in a string.

If your JSON contains "adf": true then I would expect [adf] to be a boolean in logstash and you can just use

if [adf] {
    mutate { add_field => {"TrueCondition" => "Test True" }
} else {
    mutate { add_field => {"FalseCondition" => "Test False" }
}

If you use output { stdout { codec => rubydebug } } then what does the [adf] field look like?

3

@Badger , Below is my configuration file

  syslog {
    port => 1555
        syslog_field => "message"
        grok_pattern => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname}%{GREEDYDATA:message}"
        }
}
filter {
        json {
                 source => "message"
      }
        mutate {
                remove_field => ["devTimeFormat","log","event","message"]
                }
        if "adf" in [message] {
                mutate {
                        add_field => {"deviceProduct" => "TEST.AVI"}
                        }
                }
        }
output {
        stdout {}
}

I am using STDOUT to check if the output is as expected.

My objective is - if the Syslog message has the field "adf", then I will perform "add_field", else I should drop the event and not process the event.
Please advise on how to configure this.

The whole code is working except for the IF condition., and I have not implemented the drop condition.

Thanks in advance.

Siddarth

4

If you want to test whether the [adf] field exists then use

if [adf] {
    mutate { ... }
} else {
    drop {}
}
5

@Badger , yes, my objective is the ADF field is present in the JSON log, then I will mutate and add_field, else I will drop the event.

But the IF condition is not working. Am I calling a Wrong field in IF?
Because when I just add IF, there is change in output, but when I add DROP condition, then the event is being sent to a different port and there by dropped.

Below is the updated configuration

syslog {
   port => 1555
       syslog_field => "message"
       grok_pattern => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname}%{GREEDYDATA:message}"
       }
}
filter {
       json {
                source => "message"
     }
       mutate {
               remove_field => ["devTimeFormat","log","event","message"]
               }
       if [adf] {
               mutate {
                       add_field => {"deviceProduct" => "TEST.AVI"}
                       }
               }
       }
output {
       stdout {}
}

Below is the STDOUT when I add DROP condition

[INFO ] 2022-08-04 19:14:46.428 [Ruby-0-Thread-16: :1] syslog - new connection {:client=>"10.1.0.4:44850"}

Below is a sample event

'Jul 29 10:28:49 192.168.153.101 {"adf": false,"significant":0,"udf":false,.....}'

--
Thanks in advance
Siddarth

6

The logstash configuration language does not provide a way to directly test whether a boolean field exists. There is an open issue for that. "if [adf]" will evaluate false is [adf] does not exists, but also when [adf] is a boolean with the value false.

The trick is to set a metadata field, then only overwrite it if [adf] exists, then test whether the metadata field was modified.

    mutate { add_field => { "[@metadata][test_field_check]" => "someValue" } }
    mutate { copy => { "adf" => "[@metadata][test_field_check]" } }
    if [@metadata][test_field_check] == "someValue" {
        mutate { add_field => { "field_did_not_exist" => true }}
    } else {
        mutate { add_field => { "field_did_exist" => true }}
    }
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.