# What is wrong in my grok filter work for logstash?

**URL:** <https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583>\
**Category:** Logstash\
**Created:** [October 21, 2016, 8:31am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583 "2016-10-21T08:31:51Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [October 21, 2016, 8:31am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/1 "2016-10-21T08:31:51Z")

</div>

I would like to create a grok filter in Logstash to set as field, a concrete part of a syslog message `TTN-GW` string should be that field, the sort of message that is being sent to Logstash is as follows ( it comes from a syslog)

`Oct 21 09:47:12 aaaa TTN-GW[1401]: #033[90m DEBUG#033[0m Handle stat #033[90mGatewayID#033[0m=SFDEGFKKGRSDFEEE #033[90madapter#033[0m=gateway-semtech`

My grok filter looks like this, I have created it modifying a syslog filter (shown below),

```
    filter {
      if [type] == "TTN" {
        grok {
    
          match => { "message" => "%{DATE:date} %{SYSLOGHOST:syslog_hostname} %{TTNSERVICE:ttn_service}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => ["TTN_Service", "%{ttn_service}"]
        }
      }
    }

```

However, it doesn't work as expected and I don't know why, any hint?

Thanks in advance!

The following grok filter for syslog parse it properly,

```
    filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => ["received_at", "%{@timestamp}"]
          add_field => ["received_from", "%{host}"]
        }
        syslog_pri { }
        date {
          match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
        }
      }
    }
```

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [October 21, 2016, 8:39am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/2 "2016-10-21T08:39:33Z")

</div>

Would somethint like this work too?

```
filter {
  if [type] == "TTN" {
    grok {
      match => { "message", "... TTN-GW ..." }
      add_tag => ["TTN_Service", "TTN-GW"]
    }
    grok {
      match => { "message", "... TTN-Node-Red ..." }
      add_tag => ["TTN_Service", "TTN-Node-Red"]
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 21, 2016, 11:30am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/3 "2016-10-21T11:30:49Z")

</div>

The DATE pattern doesn't match your timestamp. Why are you using it instead of SYSLOGTIMESTAMP which worked?

> ```
> add_field => ["TTN_Service", "%{ttn_service}"]
> 
> ```

Why not capture straight into the `TTN_Service` field instead of capturing into `ttn_service` and copying that value to `TTN_Service`?

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [October 26, 2016, 7:07am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/4 "2016-10-26T07:07:15Z")

</div>

> [@magnusbaeck](#):
>
> The DATE pattern doesn't match your timestamp. Why are you using it instead of SYSLOGTIMESTAMP which worked?

You were right @magnusbaeck, it was the timestamp, I updated it and now it works.

> [@magnusbaeck](#):
>
> Why not capture straight into the TTN\_Service field instead of capturing into ttn\_service and copying that value to TTN\_Service?

Sorry I am a bit newbie in this, how do you mean?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 26, 2016, 7:09am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/5 "2016-10-26T07:09:11Z")

</div>

> Sorry I am a bit newbie in this, how do you mean?

This is unnecessarily complicated:

```plaintext
grok {
  match => {
    "message" => "%{DATE:date} %{SYSLOGHOST:syslog_hostname} %{TTNSERVICE:ttn_service}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
  }
  add_field => ["TTN_Service", "%{ttn_service}"]
}

```

It can be replaced by this:

```plaintext
grok {
  match => {
    "message" => "%{DATE:date} %{SYSLOGHOST:syslog_hostname} %{TTNSERVICE:TTN_Service}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [October 26, 2016, 7:14am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/6 "2016-10-26T07:14:22Z")

</div>

Ok, I will try it

Just for the sake of learning, why is the first option more complicated?

Thanks!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 26, 2016, 7:25am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/7 "2016-10-26T07:25:31Z")

</div>

Because you have an unnecessary `add_field` and you're creating both `TTN_Service` and `ttn_service`.

---

<div class="post-metadata">

**Author:** ![ndarkness](https://avatars.discourse-cdn.com/v4/letter/n/90db22/32.png) [@ndarkness](https://discuss.elastic.co/u/ndarkness)\
**Post date:** [October 26, 2016, 7:26am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/8 "2016-10-26T07:26:46Z")

</div>

Understood, thanks 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:32am UTC](https://discuss.elastic.co/t/what-is-wrong-in-my-grok-filter-work-for-logstash/63583/9 "2017-07-06T04:32:44Z")

</div>