Logstash filter doesn't return correct results based on conditionals

[nmoham (Naveed)]

I've have the following conf file in logstash; based on the incoming data from beats containing the environment field, I want to add a new field "[@metadata][shared_env_category]".

I am testing the it using stdin and with sample data environment: prod; expected results shared_env_category => "prod", but it returns shared_env_category => "nonprod"

Logstash conf -

input { stdin { } }


filter {


mutate {
 id => "normalize_data"
 add_field => { "[top][ingest_method]" => "sysbeats" }
 add_field => { "[@metadata][lc_environment]" => "%{environment}" }
 add_field => { "[@metadata][index]" => "%{[@metadata][beat]}" }
}

# Promote an index field if exists
if [index] and [index] != "" {
 mutate {
   update => { "[@metadata][index]" => "%{index}" }
   remove_field => [ "index" ]
 }
}

mutate {
 id => "lowercase_data"
 lowercase => [ "[@metadata][lc_environment]" ]
 lowercase => [ "[@metadata][index]" ]
}

# split our data into 2 categories (prod/nonprod) for retentions
if [@metadata][lc_environment] != "prod" {
 mutate {
   id => "add_env_category_nonprod"
   add_field => { "[@metadata][shared_env_category]" => "nonprod" }
 }
} else {
 mutate {
   id => "add_env_category_prod"
   add_field => { "[@metadata][shared_env_category]" => "prod" }
 }
}
}


output { stdout { codec => rubydebug { metadata => true } } }

Tested Output (Hostname masked)

./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/beats.conf
"environment": "prod"

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
    "@timestamp" => 2020-08-08T02:30:41.463Z,
           "top" => {
        "ingest_method" => "sysbeats"
    },
     "@metadata" => {
                      "index" => "%{[@metadata][beat]}",
             "lc_environment" => "%{environment}",
        "shared_env_category" => "nonprod"
    },
          "host" => "somehost",
      "@version" => "1",
       "message" => "\"environment\": \"prod\""
}

Expected output based on std input-

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
    "@timestamp" => 2020-08-08T02:30:41.463Z,
           "top" => {
        "ingest_method" => "sysbeats"
    },
     "@metadata" => {
                      "index" => "%{[@metadata][beat]}",
             "lc_environment" => "%{environment}",
        "shared_env_category" => "prod"
    },
          "host" => "somehost",
      "@version" => "1",
       "message" => "\"environment\": \"prod\""
}

Not sure, where I am messing up. Any help or guidance appreciated.

[Jenni]

Your test scenario is based on the assumption that your input contains a field called environment. But it doesn't. Your input is a field called message with the content "environment":"prod". That doesn't automatically create a new field. It's just a string. To turn this into a flexible test system that works similarly to what you had expected with your current setup, you could instead feed it input like {"environment":"prod"} and parse that with a JSON filter. But make sure that your real data will actually contain that field

[nmoham (Naveed)]

Thanks @Jenni

I realized it and added -

input { stdin { codec => json } }

Then used {"environment":"prod"} and similar values to test successfully.