LogStash and parsing OPNSenser logs

[LoggingJennfier (Jennifer)]

My logs are coming in as follows:

<134>May 24 14:39:32 edge.internal filterlog[2535]: 78,,,ffe6d10d1f27a42fc0edc3abb3a6d333,ovpnc1,match,pass,out,4,0x0,,63,61951,0,DF,6,tcp,60,10.8.0.2,20.44.17.5,44575,443,0,S,1497081603,,64240,,mss;sackOK;TS;nop;wscale

LogStash is correctly collecting the logs but the patterns is not working for some reason.

They should be matching on OPNSENSE

My inputs.conf:

input {
  ### Firewall ###
  syslog {
    id => "pfelk-firewall-0001"
    type => "firewall"
    port => 5140
    syslog_field => "message"
    ecs_compatibility => v1
#    grok_pattern => "<%{POSINT:[log][syslog][priority]}>%{GREEDYDATA:pfelk}"
    grok_pattern => "%{GREEDYDATA:pfelk}"
    #ssl => true
    #ssl_certificate_authorities => ["/etc/logstash/ssl/YOURCAHERE.crt"]
    #ssl_certificate => "/etc/logstash/ssl/SERVER.crt"
    #ssl_key => "/etc/logstash/ssl/SERVER.key"
    #ssl_verify_mode => "force_peer"
    tags => ["pfelk"]
  }
}
#
filter {
  grok {
    patterns_dir => [ "/etc/logstash/conf.d/patterns/" ]
    match => [ "pfelk", "%{PFELK}" ]
  }
#### RFC 5424 Date/Time Format ####
  date {
    match => [ "[event][created]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
    target => "[event][created]"
  }
}

apps.conf:

...
...
  ### filterlog ###
  if [log][syslog][appname] =~ /^filterlog$/ {
    mutate {
      add_tag => "firewall"
      add_field => { "[ecs][version]" => "1.7.0" }
      add_field => { "[event][dataset]" => "pfelk.firewall" }
      replace => { "[log][syslog][appname]" => "firewall" }
    }
    grok {
      patterns_dir => [ "/etc/logstash/conf.d/patterns/" ]
      match => [ "filter_message", "%{PF_LOG_ENTRY}" ]
    }
    if [network][direction] =~ /^out$/ {
      mutate {
        rename => { "[pf][transport][data_length]" => "[destination][bytes]" }
      }
    }
    if [network][direction] =~ /^in$/ {
      mutate {
        rename => { "[pf][transport][data_length]" => "[source][bytes]" }
      }
    }
  }
...
...

patterns.pfelk:

PFELK (%{PFSENSE}|%{OPNSENSE})

# pfSense
PFSENSE (%{PFSENSE_LOG}|%{PFSENSE5424_LOG})
PFSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][app>
PFSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s(%{SYSLOGHOST:[log][syslog][hostname]}\s)?%{PROG:[log][syslog][appname]}(\[%{POSINT:[log][syslog][proci>

# OPNsense
OPNSENSE (%{OPNSENSE_LOG}|%{OPNSENSE5424_LOG})
OPNSENSE5424_LOG (%{INT:[log][syslog][version]}\s*)%{TIMESTAMP_ISO8601:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][ap>
OPNSENSE_LOG %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}\[%{POSINT:[log][syslog][procid]}>
...
...

I'm pretty sure the issue has something to do with a match failing but I'm not sure where or how to properly debug this more.

[LoggingJennfier (Jennifer)]

I found this:

OPNSENSE_DIRECTION \bin(?:/out)?|out\b

OPNSENSE_WORD \b[\w-]+\b

OPNSENSE_FILTERLOG %{OPNSENSE_TCP}|%{OPNSENSE_UDP}

OPNSENSE_IPV4 %{OPNSENSE_PACKETFILTER},%{INT:ip_ver},%{BASE16NUM:tos},%{OPNSENSE_WORD:ecn}?,%{INT:ttl},%{INT:id},%{INT:offset},%{OPNSENSE_WORD:flags},%{INT:proto_num},%{OPNSENSE_WORD:proto},%{INT:length},%{IPV4:src},%{IPV4:dst}

OPNSENSE_IPV6 %{OPNSENSE_PACKETFILTER},%{INT:ip_ver},%{BASE16NUM:class},%{BASE16NUM:flow},%{INT:hop_limit},%{OPNSENSE_WORD:proto},%{INT:proto_num},%{INT:length},%{IPV6:src},%{IPV6:dst}

OPNSENSE_LABEL (?:\b%{OPNSENSE_WORD:label}\b|\(null\))

OPNSENSE_PACKETFILTER %{INT:rule},%{INT:subrule}?,%{INT:anchor}?,%{OPNSENSE_LABEL:label},%{OPNSENSE_WORD:interface},%{OPNSENSE_WORD:reason},%{OPNSENSE_WORD:action},%{OPNSENSE_DIRECTION:dir}

OPNSENSE_TCP (?:%{OPNSENSE_IPV4}|%{OPNSENSE_IPV6}),%{INT:src_port},%{INT:dst_port},%{INT:data_len},%{OPNSENSE_WORD:tcp_flags},(?:%{INT:tcp_seq}(?:\:\d+)?)?,%{INT:tcp_ack}?,%{INT:tcp_win},%{INT:tcp_urg}?,(?<tcp_options>[^,]+)?

OPNSENSE_UDP (?:%{OPNSENSE_IPV4}|%{OPNSENSE_IPV6}),%{INT:src_port},%{INT:dst_port},%{INT:data_len}

and I'm getting some parsing:

I'm guessing this is more in regards to log format layout now?

[Lemonhard]

Hi Jennifer,
i'm working with pfelk since few day, i forked it and adapted to my custom ELK stack.
Everything seems to be ok with pfsense (i have no OPNSense around) except few fixes with the grok debugger, i would suggest you to do the same :

1. From Kibana, open the grok debugger ( Manage > Dev Tools > Grok debugger)
2. Copy a full message sample (from Discover, event.original field) and paste it into the first input box.
3. As Grok pattern, simply start with %{PFELK}
4. Paste the whole file patterns.pfelk as is into the custom patterns input box
5. Click Simulate and you will have the result if it matchs.
6. If not, reduce into sub patterns until you find the not matching one (ex: %{OPNSENSE5424_LOG} )

you may encounter this error as me :
[patterns] Invalid regex pattern found in: [%{PFELK}]. pattern [PF_PROTOCOL_DATA] is referencing a non-existent pattern [PF_IPv6_ICMP]

Using the lastest version of the pattern file, a pattern is blank and it seems not a problem for Logstash but Grok debugger... Simply remove |%{PF_IPv6_ICMP} at the end of line 76

Hope it will help,
Regards,