# LogStash and parsing OPNSenser logs

**URL:** <https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234>\
**Category:** Logstash\
**Created:** [May 24, 2023, 3:12pm UTC](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234 "2023-05-24T15:12:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![LoggingJennfier](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@LoggingJennfier](https://discuss.elastic.co/u/LoggingJennfier)\
**Post date:** [May 24, 2023, 3:12pm UTC](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234/1 "2023-05-24T15:12:27Z")

</div>

My logs are coming in as follows:

```auto
<134>May 24 14:39:32 edge.internal filterlog[2535]: 78,,,ffe6d10d1f27a42fc0edc3abb3a6d333,ovpnc1,match,pass,out,4,0x0,,63,61951,0,DF,6,tcp,60,10.8.0.2,20.44.17.5,44575,443,0,S,1497081603,,64240,,mss;sackOK;TS;nop;wscale

```

LogStash is correctly collecting the logs but the patterns is not working for some reason.

They should be matching on OPNSENSE

My inputs.conf:

```auto
input {
  ### Firewall ###
  syslog {
    id => "pfelk-firewall-0001"
    type => "firewall"
    port => 5140
    syslog_field => "message"
    ecs_compatibility => v1
# grok_pattern => "<%{POSINT:[log][syslog][priority]}>%{GREEDYDATA:pfelk}"
    grok_pattern => "%{GREEDYDATA:pfelk}"
    #ssl => true
    #ssl_certificate_authorities => ["/etc/logstash/ssl/YOURCAHERE.crt"]
    #ssl_certificate => "/etc/logstash/ssl/SERVER.crt"
    #ssl_key => "/etc/logstash/ssl/SERVER.key"
    #ssl_verify_mode => "force_peer"
    tags => ["pfelk"]
  }
}
#
filter {
  grok {
    patterns_dir => ["/etc/logstash/conf.d/patterns/"]
    match => ["pfelk", "%{PFELK}"]
  }
#### RFC 5424 Date/Time Format ####
  date {
    match => ["[event][created]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
    target => "[event][created]"
  }
}

```

apps.conf:

```auto
...
...
  ### filterlog ###
  if [log][syslog][appname] =~ /^filterlog$/ {
    mutate {
      add_tag => "firewall"
      add_field => { "[ecs][version]" => "1.7.0" }
      add_field => { "[event][dataset]" => "pfelk.firewall" }
      replace => { "[log][syslog][appname]" => "firewall" }
    }
    grok {
      patterns_dir => ["/etc/logstash/conf.d/patterns/"]
      match => ["filter_message", "%{PF_LOG_ENTRY}"]
    }
    if [network][direction] =~ /^out$/ {
      mutate {
        rename => { "[pf][transport][data_length]" => "[destination][bytes]" }
      }
    }
    if [network][direction] =~ /^in$/ {
      mutate {
        rename => { "[pf][transport][data_length]" => "[source][bytes]" }
      }
    }
  }
...
...

```

patterns.pfelk:

```auto
PFELK (%{PFSENSE}|%{OPNSENSE})

# pfSense
PFSENSE (%{PFSENSE_LOG}|%{PFSENSE5424_LOG})
PFSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][app>
PFSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s(%{SYSLOGHOST:[log][syslog][hostname]}\s)?%{PROG:[log][syslog][appname]}(\[%{POSINT:[log][syslog][proci>

# OPNsense
OPNSENSE (%{OPNSENSE_LOG}|%{OPNSENSE5424_LOG})
OPNSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][ap>
OPNSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\[%{POSINT:[log][syslog][procid]}>
...
...

```

I'm pretty sure the issue has something to do with a match failing but I'm not sure where or how to properly debug this more.

---

<div class="post-metadata">

**Author:** ![LoggingJennfier](https://avatars.discourse-cdn.com/v4/letter/l/b5e925/32.png) [@LoggingJennfier](https://discuss.elastic.co/u/LoggingJennfier)\
**Post date:** [May 24, 2023, 3:39pm UTC](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234/2 "2023-05-24T15:39:44Z")

</div>

I found this:

```auto
OPNSENSE_DIRECTION \bin(?:/out)?|out\b

OPNSENSE_WORD \b[\w-]+\b

OPNSENSE_FILTERLOG %{OPNSENSE_TCP}|%{OPNSENSE_UDP}

OPNSENSE_IPV4 %{OPNSENSE_PACKETFILTER},%{INT:ip_ver},%{BASE16NUM:tos},%{OPNSENSE_WORD:ecn}?,%{INT:ttl},%{INT:id},%{INT:offset},%{OPNSENSE_WORD:flags},%{INT:proto_num},%{OPNSENSE_WORD:proto},%{INT:length},%{IPV4:src},%{IPV4:dst}

OPNSENSE_IPV6 %{OPNSENSE_PACKETFILTER},%{INT:ip_ver},%{BASE16NUM:class},%{BASE16NUM:flow},%{INT:hop_limit},%{OPNSENSE_WORD:proto},%{INT:proto_num},%{INT:length},%{IPV6:src},%{IPV6:dst}

OPNSENSE_LABEL (?:\b%{OPNSENSE_WORD:label}\b|\(null\))

OPNSENSE_PACKETFILTER %{INT:rule},%{INT:subrule}?,%{INT:anchor}?,%{OPNSENSE_LABEL:label},%{OPNSENSE_WORD:interface},%{OPNSENSE_WORD:reason},%{OPNSENSE_WORD:action},%{OPNSENSE_DIRECTION:dir}

OPNSENSE_TCP (?:%{OPNSENSE_IPV4}|%{OPNSENSE_IPV6}),%{INT:src_port},%{INT:dst_port},%{INT:data_len},%{OPNSENSE_WORD:tcp_flags},(?:%{INT:tcp_seq}(?:\:\d+)?)?,%{INT:tcp_ack}?,%{INT:tcp_win},%{INT:tcp_urg}?,(?<tcp_options>[^,]+)?

OPNSENSE_UDP (?:%{OPNSENSE_IPV4}|%{OPNSENSE_IPV6}),%{INT:src_port},%{INT:dst_port},%{INT:data_len}

```

and I'm getting some parsing:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/5/c5be977e211e687087cc5d68c71b5a2707a83f07.png)

I'm guessing this is more in regards to log format layout now?

---

<div class="post-metadata">

**Author:** ![Lemonhard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lemonhard/32/122498_2.png) [@Lemonhard](https://discuss.elastic.co/u/Lemonhard)\
**Post date:** [June 20, 2023, 9:16am UTC](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234/3 "2023-06-20T09:16:07Z")

</div>

Hi Jennifer,  
i'm working with pfelk since few day, i forked it and adapted to my custom ELK stack.  
Everything seems to be ok with pfsense (i have no OPNSense around) except few fixes with the grok debugger, i would suggest you to do the same :

1. From Kibana, open the grok debugger ( Manage \> Dev Tools \> Grok debugger)
2. Copy a full message sample (from Discover, event.original field) and paste it into the first input box.
3. As Grok pattern, simply start with `%{PFELK}`
4. Paste the whole file patterns.pfelk as is into the custom patterns input box
5. Click Simulate and you will have the result if it matchs.
6. If not, reduce into sub patterns until you find the not matching one (ex: `%{OPNSENSE5424_LOG}` )

you may encounter this error as me :  
[patterns] Invalid regex pattern found in: [%{PFELK}]. pattern [PF\_PROTOCOL\_DATA] is referencing a non-existent pattern [PF\_IPv6\_ICMP]

Using the [lastest version of the pattern file](https://github.com/pfelk/pfelk/blob/main/etc/pfelk/patterns/pfelk.grok), a pattern is blank and it seems not a problem for Logstash but Grok debugger... Simply remove `|%{PF_IPv6_ICMP}` at the end of line 76

 ![2023-06-20 11_14_15-Window](https://us1.discourse-cdn.com/elastic/original/3X/2/1/214f7fcc1df9817cb027870b633cf1cec651cb8d.png)

Hope it will help,  
Regards,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2023, 9:17am UTC](https://discuss.elastic.co/t/logstash-and-parsing-opnsenser-logs/334234/4 "2023-07-18T09:17:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.