Logstash KV filter not working on Greedydata

Elastic Stack Logstash
1

Raw logs

<190>SCO-0000-CS01: 2025 Apr 3 17:53:36 UTC: %ACLLOG-6-ACLLOG: SGT: 0, Src IP: 1.1.1.1, Dst IP: 9.9.9.9, Src Port: 504, Dst Port: 443, Src Intf: port-channel1002, Protocol: "UDP"(17), ACL Name: alltraffic, ACE Action: Permit, Appl Intf: Vlan16, Hit-count: 1

The KV on the rest on the greedydata is not processing


filter {


 dissect {
      mapping => {
        "message" => "<%{INT:syslog_priority}>%{WORD:device_location}-%{WORD:device_model}-%{WORD:device_id}: %{YEAR:year} %{MONTH:month}  %{MONTHDAY:day} %{TIME:time} %{WORD:timezone}: %%{DATA:syslog_message_type}: %{GREEDYDATA:restOfLine}"
      }
    }
    kv {
      source => "restOfLine"
      field_split => ","
      trim_key => " "
      value_split => ":"
    }


}
2

You cannot use dissect in that way. That is the grok syntax. It should be like this:
grok { match => { "message" => "<%{INT:syslog_priority}>%{WORD:device_location}-%{WORD:device_model}-%{WORD:device_id}: %{YEAR:year} %{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{WORD:timezone}: %%{DATA:syslog_message_type}: %{GREEDYDATA:restOfLine}" } }

PS. You had double spaces in front of %{MONTHDAY:day}

3

the raw message has 2 spaces after the Month , that was intended ! , but is there a way to keep this part together

SCO-N000-CS01 as hostname and its not parsing at all now


     grok { match => { "message" => "<%{INT:syslog_priority}>%{WORD:device_location}-%{WORD:device_model}-%{WORD:device_id}: {YEAR:year} %{MONTH:month}  %{MONTHDAY:day} %{TIME:time} %{WORD:timezone}: %%{DATA:syslog_message_type}: %{GREEDYDATA:restOfLine}" } }
    kv {
      source => "restOfLine"
      field_split => ","
      trim_key => " "
      value_split => ":"
    }


#}
}
4

Of course. You can do it with grok, or you can do it with dissect. With grok you could try

    grok {
        pattern_definitions => {
            "MYTIMESTAMP" => "%{YEAR} %{SYSLOGTIMESTAMP} %{WORD:timezone}"
            "MYHOSTNAME" => "%{WORD}-%{WORD}-%{WORD}"
        }
        match => { "message" => "<%{INT:syslog_priority}>%{MYHOSTNAME:hostname}: %{MYTIMESTAMP:timestamp}: %%{DATA:syslog_message_type}-%{INT:severity}-%{WORD:log_message}: %{GREEDYDATA:restOfLine}" }
    }

with dissect you could try

dissect { mapping => { "message" => "<%{priority}>%{hostname}: %{timestamp}: %{msgcode}: %{restOfLine}" } }

Either way the kv will produce

 "Hit-count" => "1",
  "Src Intf" => "port-channel1002",
    "Dst IP" => "9.9.9.9",
  "Protocol" => "\"UDP\"(17)",
    "Src IP" => "1.1.1.1",
       "SGT" => "0",
  "Dst Port" => "443",

etc.

Your grok pattern has to match either Mar 9 with two spaces or Mar 10 with one. The grok SYSLOGTIMESTAMP pattern does this.

1 Like
5

Just to add:

  • in case you have variable size delimiters or optional fields go for grok otherwise dissect is much faster
  • don't forget to convert datetime to the date format otherwise you will have datetime as string.