# No Logstash errors, but no data

**URL:** <https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971>\
**Category:** Logstash\
**Created:** [March 5, 2019, 6:34pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971 "2019-03-05T18:34:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnfowles](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@johnfowles](https://discuss.elastic.co/u/johnfowles)\
**Post date:** [March 5, 2019, 6:34pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/1 "2019-03-05T18:34:21Z")

</div>

Working indexing and adding fields to OpenVPN logs in pfsense, the original config (stolen from git) works fine and parses out all the pfsense syslog inputs correctly.  
When I add in the following I get no errors in the Logstash logs, but no more data shows up. Not sure how to track down the issue(s).

```
grok {
    patterns_dir => ["/etc/logstash/conf.d/patterns"]
    match => ["message", "%{PFSENSE_OPENVPN}"]
     }
      if ![geoip] and [src_ip] !~ /^(10\.|192\.168\.)/ {
          geoip {
               add_tag => ["GeoIP"]
               source => "src_ip"
               database => "/etc/logstash/geoipdbs/GeoLite2-City.mmdb"
      }
   }

```

With the added grok pattern of  
`PFSENSE_OPENVPN ((%{USERNAME:username}[/]%{IPV4:src_ip}:%{INT:src_port})|(%{IPV4:src_ip}:%{INT:src_port})) %{GREEDYDATA:syslog_message}`

---

<div class="post-metadata">

**Author:** ![johnfowles](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@johnfowles](https://discuss.elastic.co/u/johnfowles)\
**Post date:** [March 5, 2019, 7:52pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/2 "2019-03-05T19:52:59Z")

</div>

I got this to work by just making a separate filter file for anything with the openvpn tag that was originally assigned. But I still get a grok failure even though the message field will correctly parse with the above grok pattern.

Still not sure why that is the case and any help/recommendations are welcome, thanks!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 5, 2019, 8:27pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/3 "2019-03-05T20:27:28Z")

</div>

You have not told us what the data looks like.

---

<div class="post-metadata">

**Author:** ![johnfowles](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@johnfowles](https://discuss.elastic.co/u/johnfowles)\
**Post date:** [March 5, 2019, 9:48pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/4 "2019-03-05T21:48:12Z")

</div>

The input is originally from syslog and the message field I am trying to match looks like

`"message": "user/9.9.9.9:27956 MULTI_sva: pool returned IPv4=10.77.7.2, IPv6=(Not enabled)"`

The syslog preprocess filter is

```
filter {
  if [type] == "syslog" {
    if [host] =~ /10\.0\.0\.1/ {
      mutate {
        add_tag => ["pfsense", "Ready"]
      }
    }
    if "Ready" not in [tags] {
      mutate {
        add_tag => ["syslog"]
      }
    }
  }
}
filter {
  if [type] == "syslog" {
    mutate {
      remove_tag => "Ready"
    }
  }
}
filter {
  if "syslog" in [tags] {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
      locale => "en"
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => ["@source_host", "%{syslog_hostname}"]
        replace => ["@message", "%{syslog_message}"]
      }
    }
    mutate {
      remove_field => ["syslog_hostname", "syslog_message", "syslog_timestamp"]
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 5, 2019, 10:03pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/5 "2019-03-05T22:03:12Z")

</div>

That message looks nothing like the pattern you are trying to match it to. A \_grokparsefailure is expected.

```
user/9.9.9.9:27956 MULTI_sva: pool returned IPv4=10.77.7.2, IPv6=(Not enabled)"`

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

```

This works. Not sure what you are trying to do with the additional groups (parentheses) and alternation (|).

```
    grok {
        pattern_definitions => {
            "PFSENSE_OPENVPN" => "^%{USERNAME:username}/%{IPV4:src_ip}:%{INT:src_port} %{GREEDYDATA:syslog_message}"
        }
        match => { "message" => "%{PFSENSE_OPENVPN}" }
    }
```

---

<div class="post-metadata">

**Author:** ![johnfowles](https://avatars.discourse-cdn.com/v4/letter/j/91b2a8/32.png) [@johnfowles](https://discuss.elastic.co/u/johnfowles)\
**Post date:** [March 5, 2019, 10:25pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/6 "2019-03-05T22:25:46Z")

</div>

Yep, that was it. I got rid of the group redundancy.  
Also may have forgotten to put the definition variable into {}....

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 2, 2019, 10:38pm UTC](https://discuss.elastic.co/t/no-logstash-errors-but-no-data/170971/7 "2019-04-02T22:38:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.