Improper behaviour of Logstash kv filter

[Aniket_Pant (kaal)]

filter {
if [type] == "rsyslog" {
if [message] {
kv {
source => "message"
field_split => ","
prefix => "syslog_"
transform_key => "lowercase"
}
}
}
}

{
  "_index": "client_syslog-2020.09.24",
  "_type": "_doc",
  "_id": "EXCWv3QBrbWnWCKdZnCl",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "host": "192.168.0.119",
    "message": "{\"@timestamp\":\"2020-09-24T15:38:58.138153+05:30\",\"@version\":\"1\",\"message\":\"pam_unix(sshd:session): session opened for user root by (uid=0)\",\"sysloghost\":\"client\",\"severity\":\"info\",\"facility\":\"authpriv\",\"programname\":\"sshd\",\"procid\":\"2852\"}",
    "@timestamp": "2020-09-24T10:08:58.575Z",
    "port": 54350,
    "type": "rsyslog",
    "syslog_\"message\":\"pam_unix(sshd:session): session opened for user root by (uid": "0)\""
  },
  "fields": {
    "@timestamp": [
      "2020-09-24T10:08:58.575Z"
    ]
  },
  "sort": [
    1600942138575
  ]
}

only syslog_message field is separated and which is not properly separated and rest of the field of is not separated

[Wolfram_Haussig]

Hi,

If I understand it right your message contains JSON, correct? Why don't you use the Json filter then?

Best regards
Wolfram

[jfs1 (Jean-François Stenuit)]

By default, the kv filter splits the field and value part with the first "=" sign. If you want to parse standard linux syslog, you'd better of using some grok filters, as they are not usually formated as kay-value pair.

A message suitable for KV parsing is something like :

timestamp=1599829393 tz="UTC+2:00" devname="Fortigate" devid="FGT4XH6I25410001" vd="FW_IT" date=2020-09-11 time=15:03:13 logid="0000000013" type="traffic" subtype="forward" level="notice" eventtime=1599829393356162487 tz="+0200" srcip=192.168.14.23 srcport=59636 srcintf="INTFWILD-01" srcintfrole="undefined" dstip=10.10.123.4 dstport=52630 dstintf="INTRANET-04" dstintfrole="undefined" sessionid=758066474 proto=17 action="deny" policyid=0 policytype="policy" service="udp/52630"dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel="high"

[Aniket_Pant (kaal)]

I am trying to parsing the message field but i can't because of '{' in message field

[Badger]

Use a json filter, not a kv filter.