How to parse Proxmox log format?

[gmbynkrkrqz]

Hello,

I would like to store all my Proxmox logs in Elasticsearch. I know there are Groks and I already use its for some other logs. However, as you can see below, the log format is often different : the fields are not placed in the same place or doesn't exists in some log lines. Then, how can I do to parse these Proxmox logs with grok patterns ?

0 7 PVEFW-HOST-OUT 01/Jul/2020:00:42:13 +0200 DROP: OUT=vmbr1 SRC=192.168.1.2 DST=192.168.1.56 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42440 DF PROTO=TCP SPT=23658 DPT=443 SEQ=2516196194 ACK=0 WINDOW=64140 SYN 
0 7 PVEFW-HOST-IN 01/Jul/2020:00:42:18 +0200 DROP: IN=vmbr0 PHYSIN=xxxx MAC=xxxxxxxxxxxxx SRC=fe80::126f:xx:xx:xx DST=ff02::1 LEN=32 TC=0 FLOWLBL=0 HOPLIMIT=1 NEXTHDR=HOPOPTS PROTO=ICMPV6 TYPE=130 CODE=0

These lines comes from /var/log/pve-firewall.log

[EDIT 1]

Another problem is that the firlds format isn't always <field_name> = <value>. As you can see below, the log line starts with the VM ID.

5632 7 tap10003i0-IN 01/Jul/2020:16:46:14 +0200 policy DROP: IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN

Thanks for your future help.

[Jenni]

I didn't know what the values mean, so the names aren't useful. But would this work for you?

dissect {
  mapping => {
    "message" => "%{a} %{b} %{c} %{ts} %{+ts} %{d}: %{kvdata}"
  }
}
kv {
  source => "kvdata"
}

---

Alternatively, if you want to keep DF and SYN as empty fields:

dissect {
  mapping => {
    "message" => "%{a} %{b} %{c} %{ts} %{+ts} %{d}: %{kvdata}"
  }
}
mutate {
  gsub => ["kvdata", " ([A-Z]+)( |$)", ' \1=" " ']
}
kv {
  source => "kvdata"
  trim_value => " "
}

{
        "kvdata" => "IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF=\" \" PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN=\" \" ",
           "SPT" => "52776",
       "PHYSOUT" => "xxxxxx",
          "host" => "##########",
            "ID" => "30879",
         "PROTO" => "TCP",
      "@version" => "1",
       "message" => "5632 7 tap10003i0-IN 01/Jul/2020:16:46:14 +0200 policy DROP: IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN",
    "@timestamp" => 2020-07-29T17:02:08.959Z,
             "d" => "policy DROP",
             "c" => "tap10003i0-IN",
             "a" => "5632",
           "OUT" => "xxxx",
           "SRC" => "192.168.45.36",
           "DST" => "192.168.45.68",
           "LEN" => "60",
           "ACK" => "0",
           "TOS" => "0x00",
           "SEQ" => "1117512169",
          "PREC" => "0x00",
           "SYN" => "",
             "b" => "7",
            "IN" => "xxxxx",
           "TTL" => "61",
            "DF" => "",
           "MAC" => "xxxxxxxxxxxxxxxx",
        "PHYSIN" => "xxxxxx",
            "ts" => "01/Jul/2020:16:46:14 +0200",
           "DPT" => "443",
        "WINDOW" => "64240"
}

[gmbynkrkrqz]

Interesting ! I will look that ! Thanks !