No Logstash errors, but no data

[johnfowles (John)]

Working indexing and adding fields to OpenVPN logs in pfsense, the original config (stolen from git) works fine and parses out all the pfsense syslog inputs correctly.
When I add in the following I get no errors in the Logstash logs, but no more data shows up. Not sure how to track down the issue(s).

grok {
    patterns_dir => ["/etc/logstash/conf.d/patterns"]
    match => [ "message", "%{PFSENSE_OPENVPN}" ]
     }
      if ![geoip] and [src_ip] !~ /^(10\.|192\.168\.)/ {
          geoip {
               add_tag => [ "GeoIP" ]
               source => "src_ip"
               database => "/etc/logstash/geoipdbs/GeoLite2-City.mmdb"
      }
   }

With the added grok pattern of
PFSENSE_OPENVPN ((%{USERNAME:username}[/]%{IPV4:src_ip}:%{INT:src_port})|(%{IPV4:src_ip}:%{INT:src_port})) %{GREEDYDATA:syslog_message}

[johnfowles (John)]

I got this to work by just making a separate filter file for anything with the openvpn tag that was originally assigned. But I still get a grok failure even though the message field will correctly parse with the above grok pattern.

Still not sure why that is the case and any help/recommendations are welcome, thanks!

[Badger]

You have not told us what the data looks like.

[johnfowles (John)]

The input is originally from syslog and the message field I am trying to match looks like

"message": "user/9.9.9.9:27956 MULTI_sva: pool returned IPv4=10.77.7.2, IPv6=(Not enabled)"

The syslog preprocess filter is

filter {
  if [type] == "syslog" {
    if [host] =~ /10\.0\.0\.1/ {
      mutate {
        add_tag => ["pfsense", "Ready"]
      }
    }
    if "Ready" not in [tags] {
      mutate {
        add_tag => [ "syslog" ]
      }
    }
  }
}
filter {
  if [type] == "syslog" {
    mutate {
      remove_tag => "Ready"
    }
  }
}
filter {
  if "syslog" in [tags] {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM  dd HH:mm:ss" ]
      locale => "en"
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
  }
}

[Badger]

That message looks nothing like the pattern you are trying to match it to. A _grokparsefailure is expected.

user/9.9.9.9:27956 MULTI_sva: pool returned IPv4=10.77.7.2, IPv6=(Not enabled)"`

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

This works. Not sure what you are trying to do with the additional groups (parentheses) and alternation (|).

    grok {
        pattern_definitions => {
            "PFSENSE_OPENVPN" => "^%{USERNAME:username}/%{IPV4:src_ip}:%{INT:src_port} %{GREEDYDATA:syslog_message}"
        }
        match => { "message" => "%{PFSENSE_OPENVPN}" }
    }

[johnfowles (John)]

Yep, that was it. I got rid of the group redundancy.
Also may have forgotten to put the definition variable into {}....