# How to parse Proxmox log format?

**URL:** <https://discuss.elastic.co/t/how-to-parse-proxmox-log-format/243089>\
**Category:** Logstash\
**Created:** [July 29, 2020, 3:39pm UTC](https://discuss.elastic.co/t/how-to-parse-proxmox-log-format/243089 "2020-07-29T15:39:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![gmbynkrkrqz](https://avatars.discourse-cdn.com/v4/letter/g/22d042/32.png) [@gmbynkrkrqz](https://discuss.elastic.co/u/gmbynkrkrqz)\
**Post date:** [July 29, 2020, 3:39pm UTC](https://discuss.elastic.co/t/how-to-parse-proxmox-log-format/243089/1 "2020-07-29T15:39:22Z")

</div>

Hello,

I would like to store all my Proxmox logs in Elasticsearch. I know there are Groks and I already use its for some other logs. However, as you can see below, the log format is often different : the fields are not placed in the same place or doesn't exists in some log lines. Then, how can I do to parse these Proxmox logs with grok patterns ?

```auto
0 7 PVEFW-HOST-OUT 01/Jul/2020:00:42:13 +0200 DROP: OUT=vmbr1 SRC=192.168.1.2 DST=192.168.1.56 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42440 DF PROTO=TCP SPT=23658 DPT=443 SEQ=2516196194 ACK=0 WINDOW=64140 SYN 
0 7 PVEFW-HOST-IN 01/Jul/2020:00:42:18 +0200 DROP: IN=vmbr0 PHYSIN=xxxx MAC=xxxxxxxxxxxxx SRC=fe80::126f:xx:xx:xx DST=ff02::1 LEN=32 TC=0 FLOWLBL=0 HOPLIMIT=1 NEXTHDR=HOPOPTS PROTO=ICMPV6 TYPE=130 CODE=0

```

These lines comes from `/var/log/pve-firewall.log`

[EDIT 1]

Another problem is that the firlds format isn't always `<field_name> = <value>`. As you can see below, the log line starts with the VM ID.

```auto
5632 7 tap10003i0-IN 01/Jul/2020:16:46:14 +0200 policy DROP: IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN

```

Thanks for your future help.

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [July 29, 2020, 4:03pm UTC](https://discuss.elastic.co/t/how-to-parse-proxmox-log-format/243089/2 "2020-07-29T16:03:43Z")

</div>

I didn't know what the values mean, so the names aren't useful. But would this work for you?

```auto
dissect {
  mapping => {
    "message" => "%{a} %{b} %{c} %{ts} %{+ts} %{d}: %{kvdata}"
  }
}
kv {
  source => "kvdata"
}

```

* * *

Alternatively, if you want to keep DF and SYN as empty fields:

```auto
dissect {
  mapping => {
    "message" => "%{a} %{b} %{c} %{ts} %{+ts} %{d}: %{kvdata}"
  }
}
mutate {
  gsub => ["kvdata", " ([A-Z]+)( |$)", ' \1=" " ']
}
kv {
  source => "kvdata"
  trim_value => " "
}

```

```auto
{
        "kvdata" => "IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF=\" \" PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN=\" \" ",
           "SPT" => "52776",
       "PHYSOUT" => "xxxxxx",
          "host" => "##########",
            "ID" => "30879",
         "PROTO" => "TCP",
      "@version" => "1",
       "message" => "5632 7 tap10003i0-IN 01/Jul/2020:16:46:14 +0200 policy DROP: IN=xxxxx OUT=xxxx PHYSIN=xxxxxx PHYSOUT=xxxxxx MAC=xxxxxxxxxxxxxxxx SRC=192.168.45.36 DST=192.168.45.68 LEN=60 TOS=0x00 PREC=0x00 TTL=61 ID=30879 DF PROTO=TCP SPT=52776 DPT=443 SEQ=1117512169 ACK=0 WINDOW=64240 SYN",
    "@timestamp" => 2020-07-29T17:02:08.959Z,
             "d" => "policy DROP",
             "c" => "tap10003i0-IN",
             "a" => "5632",
           "OUT" => "xxxx",
           "SRC" => "192.168.45.36",
           "DST" => "192.168.45.68",
           "LEN" => "60",
           "ACK" => "0",
           "TOS" => "0x00",
           "SEQ" => "1117512169",
          "PREC" => "0x00",
           "SYN" => "",
             "b" => "7",
            "IN" => "xxxxx",
           "TTL" => "61",
            "DF" => "",
           "MAC" => "xxxxxxxxxxxxxxxx",
        "PHYSIN" => "xxxxxx",
            "ts" => "01/Jul/2020:16:46:14 +0200",
           "DPT" => "443",
        "WINDOW" => "64240"
}

```

---

<div class="post-metadata">

**Author:** ![gmbynkrkrqz](https://avatars.discourse-cdn.com/v4/letter/g/22d042/32.png) [@gmbynkrkrqz](https://discuss.elastic.co/u/gmbynkrkrqz)\
**Post date:** [July 30, 2020, 12:04pm UTC](https://discuss.elastic.co/t/how-to-parse-proxmox-log-format/243089/3 "2020-07-30T12:04:50Z")

</div>

Interesting ! I will look that ! Thanks !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2020, 12:04pm UTC](https://discuss.elastic.co/t/how-to-parse-proxmox-log-format/243089/4 "2020-08-27T12:04:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.