I would like to store all my Proxmox logs in Elasticsearch. I know there are Groks and I already use its for some other logs. However, as you can see below, the log format is often different : the fields are not placed in the same place or doesn't exists in some log lines. Then, how can I do to parse these Proxmox logs with grok patterns ?
0 7 PVEFW-HOST-OUT 01/Jul/2020:00:42:13 +0200 DROP: OUT=vmbr1 SRC=192.168.1.2 DST=192.168.1.56 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42440 DF PROTO=TCP SPT=23658 DPT=443 SEQ=2516196194 ACK=0 WINDOW=64140 SYN 0 7 PVEFW-HOST-IN 01/Jul/2020:00:42:18 +0200 DROP: IN=vmbr0 PHYSIN=xxxx MAC=xxxxxxxxxxxxx SRC=fe80::126f:xx:xx:xx DST=ff02::1 LEN=32 TC=0 FLOWLBL=0 HOPLIMIT=1 NEXTHDR=HOPOPTS PROTO=ICMPV6 TYPE=130 CODE=0
These lines comes from
Another problem is that the firlds format isn't always
<field_name> = <value>. As you can see below, the log line starts with the VM ID.
5632 7 tap10003i0-IN 01/Jul/2020:16:46:14 +0200 policy DROP: IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN
Thanks for your future help.