If statement not working as expected

[Mistral (Marius)]

Hi guys, I know this topic has been posted a few times, but I'm unable to find any tracks to help me since a few days.

I'm trying to adapt the logstash 5.8 configuration provided by stormshield to a higher (latest) version of the ELK stack, but I'm struggling with an if statement not working as expected.

Here's an example of my stdout rubydebug :

{
             "origdst" => "XXX.XXX.XXX.XXX",
             "ipproto" => "tcp",
             "srcport" => "55555",
         "srcportname" => "some_name",
                 "dst" => "XXX.XXX.XXX.XXX",
            "@version" => "1",
              "action" => "pass",
                 "ipv" => "4",
               "proto" => "ssl",
    "internal_product" => "%{type}",
             "dstport" => "4444",
                "rcvd" => "4252",
             "message" => "1 2022-12-12T15:48:00+01:00 SNXXXXXXXXXXXXX asqd - - - id=firewall time=\"2022-12-12 15:48:00\" fw=\"SNXXXXXXXXXXXXX\" tz=+0100 startime=\"2022-12-12 15:47:59\" pri=5 confid=00 slotlevel=2 ruleid=99 rulename=\"232e63534sa23\" srcif=\"vlan999\" srcifname=\"some_network\" ipproto=tcp dstif=\"vlan999\" dstifname=\"some_other_network\" proto=ssl src=XXX.XXX.XXX.XXX srcport=55555 srcportname=something srcname=some_name srcmac=XX:XX:XX:XX:XX:XX dst=XXX.XXX.XXX.XXX dstport=4444 dstportname=some_name dstname=some_name_again dstcontinent=\"st\" dstcountry=\"st\" modsrc=XXX.XXX.XXX.XXX modsrcport=44444 origdst=XXX.XXX.XXX.XXX origdstport=4444 ipv=4 sent=1585 rcvd=1396 duration=0.00 action=pass logtype=\"connection\"",
           "srcifname" => "some_name",
                "host" => {
              "ip" => "127.0.0.1",
        "hostname" => "XXX.XXX.XXX.XXX"
    },
             "srcname" => "some_source_name",
             "dstname" => "some_dst_name",
              "confid" => "00",
            "duration" => "0.00",
            "rulename" => "7162hjjH2873",
              "srcmac" => "XX:XX:XX:XX:XX:XX",
              "modsrc" => "XXX.XXX.XXX.XXX",
            "startime" => "2022-12-12 15:47:59",
               "event" => {
        "original" => "<14>Dec 12 15:47:59 10.63.65.250 1 2022-12-12T15:48:00+01:00 SNXXXXXXXXXXX asqd - - - id=firewall time=\"2022-12-12 15:48:00\" fw=\"SNXXXXXXXXXXX\" tz=+0100 startime=\"2022-12-12 15:47:59\" pri=5 confid=00 slotlevel=2 ruleid=99 rulename=\"7162hjjH2873\" srcif=\"vlan999\" srcifname=\"some_if_name\" ipproto=tcp dstif=\"vlan99\" dstifname=\"some_dist_name\" proto=ssl src=XXX.XXX.XXX.XXX srcport=55555 srcportname=some_port_name srcname=some_src_name srcmac=XX:XX:XX:XX:XX dst=XXX.XXX.XXX.XXX dstport=5555 dstportname=some_port_name dstname=some_dst_name dstcontinent=\"st\" dstcountry=\"st\" modsrc=XXX.XXX.XXX.XXX modsrcport=55555 origdst=XXX.XXX.XXX.XXX origdstport=4444 ipv=4 sent=1585 rcvd=1396 duration=0.00 action=pass logtype=\"connection\"\n"
    },
                  "tz" => "+0100",
                  "fw" => "SNXXXXXXXXXXX",
             "logtype" => "connection",
        "dstcontinent" => "st",
                "time" => "2022-12-12 15:48:00",
                 "id" => "firewall",
           "dstifname" => "some_dst_name",
          "modsrcport" => "55555",
          "@timestamp" => 2022-12-12T14:47:59.000Z,
             "service" => {
        "type" => "system"
    },
         "dstportname" => "some_name",
                 "pri" => "5",
           "slotlevel" => "2",
              "ruleid" => "99",
               "dstif" => "vlan99",
          "dstcountry" => "st",
         "origdstport" => "4444",
                 "src" => "XXX.XXX.XXX.XXX",
               "srcif" => "vlan999",
                "sent" => "1585",
                 "log" => {
        "syslog" => {
            "severity" => {
                "code" => 6,
                "name" => "Informational"
            },
            "priority" => 14,
            "facility" => {
                "code" => 1,
                "name" => "user-level"
            }
        }
    }
}

Please do not focus on the data I've randomized.
I've highlighted two fields causing me problems. The first one "internal_product" is supposed to take the value of the "type" field (I'm not talking about the type field contained in the service field), which does not exist probably because off the "id" field (which is not aligned with the rest, I don't know if that matters but it has triggered me).
The creation of the type field depends on the id field in the filter file, as shown below :

  if [id] == "firewall" {
    mutate {
      add_field => { "type" => "sns" }
    }
  } else if .....

  }

  mutate {
    add_field => { "internal_product" => "%{type}" }
  }

(I've put only the part I think that matters, but ask me for the rest if needed)

but this condition is not getting verified even though I have an id field corresponding to "firewall". I've tried the filter without the if statement and it works fine.
I've also tried different syntaxes (if "firewall" in [id] // if [id] =~ "^firewall" // if [id] =~ /firewall/) but none of them works.

So I'm quite lost, I'd be very gratefull for any help. If you need any others infos please let me know.

Regards,
Marius

[leandrojmp (Leandro Pereira)]

Can you share the rest of your pipeline?

From what you shared, it should've work.

[Mistral (Marius)]

Hey

Here is the complete 11-filter-standard.conf file :

# To test configuration
# /opt/logstash/bin/logstash -f /data/etc/logstash/svc/ --configtest
#
# To Debug plugins
# Uncomment "stdout { codec => rubydebug }" in out section
# Then call
# /opt/logstash/bin/logstash -e 'input { stdin{} } filter { kv {source => 'message' } SDS {} } output {stdout { codec => rubydebug }}'
# And paste lines on prompt (stdin)

filter {

  # kv filter expects non null value (like dst= dstname=foo)
  # Hence, we have to replace null value with empty quotes (dst= is replaced by dst="")
  mutate {
      gsub => ["message", '= ', '=\"\" ']
  }

  # Known issue in logstash (https://github.com/elastic/logstash/issues/1645) is related to escape double quotes
  # Here, we replace every \" with '
  # For example: "key: value1 \"value2\"" is replaced with "key: value1 'value2'"
  mutate {
      gsub => ["message", '\\"', "'"]
  }

  # Remove leading and trailing whitespaces (including newline)
  mutate {
    strip => "message"
  }

  # Remove leading and trailing quotes
  kv {
    source => "message"
    trim_value => '"\''
  }

  # Sanity check for 'id' existency
  if [path] {
    if ![id] {
      if "sns" in [path] {
        mutate {
          add_field => { "id" => "firewall" }
        }
      } else if "sdmc" in [path] {
        mutate {
          add_field => { "id" => "sdmc" }
        }
      } else if "sds" in [path] {
        mutate {
          add_field => { "id" => "datasecurity" }
        }
      } else if "ses" in [path] {
        mutate {
          add_field => { "id" => "endpoint" }
        }
      }
    }
  }

  if [id] == "firewall" {
    mutate {
      add_field => { "type" => "sns" }
    }
  } else if [id] == "endpoint" {
    mutate {
      add_field => { "type" => "ses" }
    }
  } else if [id] == "datasecurity" {
      mutate {
        add_field => { "type" => "sds" }
      }
  } else if [id] == "sdmc" {
    # if log is a policy one, it contains the 'type' field
    if [entity] == "policies" {
      mutate {
        rename => { "type" => "policy_type" }
      }
    }
    mutate {
      add_field => { "type" => "sdmc" }
    }
  }

  if [deviceVendor] == "Panda Security" {
    mutate {
      add_field => { "type" => "panda" }
    }
  }

  mutate {
    add_field => { "internal_product" => "%{type}" }
  }

  # drop events that are not identified as stormshield products
#  if [internal_product] not in ["sns", "ses", "sds", "sdmc", "panda"] {
#    drop { }
#  }
}

I've commented the last drop part because I wasn't having any output otherwise (since the "internal_product" wasn't filled properly).

Thank you,
Marius

[Mistral (Marius)]

Hey ! I'm upping this topic since I still haven't found a way to make this if statement works.