# Gork filter with custom pattern

**URL:** <https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919>\
**Category:** Logstash\
**Created:** [November 14, 2019, 3:07pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919 "2019-11-14T15:07:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rp346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rp346/32/53882_2.png) [@rp346](https://discuss.elastic.co/u/rp346)\
**Post date:** [November 14, 2019, 3:07pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919/1 "2019-11-14T15:07:46Z")

</div>

I am trying to aggregate outside logs into ElasticSearch cluster running in Kubernetes.

Syslogs get published to logstash instance which forward to elasticsearch instance. Syslogs has pattern `Timestamp Serial_Number:Process Level Message`

Sample :  
`Oct 11 21:59:02 e3ed8a4e-530f-4658-b39d-049f38cda310:systemd DEBUG waiting for reply from 10.3.1.204:123 (10.3.1.204).`

Using [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) I was able to create following filter

```
filter {
    grok {
      match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} (?<host>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}):(?<process>(?<=:).*) %{LOGLEVEL:loglevel} %{GREEDYDATA:body}"}
    }
    mutate {
        add_field => { "process" => "%{process}" }
        add_field => { "loglevel" => "%{loglevel}" }
      }
    date {
        match => ["logdate", "MMM dd HH:mm:ss"]
      }
}

```

But new added fields `process` & `loglevel` doesn't have the actual value.

Its shows up just like **process: %{process}** & **loglevel: %{loglevel}** in kibana

What is the correct filter pattern I need to use here ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 14, 2019, 5:56pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919/2 "2019-11-14T17:56:41Z")

</div>

> [@rp346](#):
>
> Its shows up just like **process: %{process}** & **loglevel: %{loglevel}** in kibana

That suggests your grok pattern is not matching, although it works for me for the sample data you showed.

Remove the mutate filter. That will just convert

```
  "loglevel" => "DEBUG",

```

into an array

```
  "loglevel" => [
    [0] "DEBUG",
    [1] "DEBUG"
],

```

---

<div class="post-metadata">

**Author:** ![rp346](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rp346/32/53882_2.png) [@rp346](https://discuss.elastic.co/u/rp346)\
**Post date:** [November 15, 2019, 9:45pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919/3 "2019-11-15T21:45:40Z")

</div>

Can you elaborate how the `filter` block would look with your suggestion ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 15, 2019, 10:13pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919/4 "2019-11-15T22:13:31Z")

</div>

```
filter {
    grok {
        match => {"message" => "%{SYSLOGTIMESTAMP:timestamp} (?<host>[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}):(?<process>(?<=:).*) %{LOGLEVEL:loglevel} %{GREEDYDATA:body}"}
    }
    date {
        match => ["logdate", "MMM dd HH:mm:ss"]
    }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2019, 10:13pm UTC](https://discuss.elastic.co/t/gork-filter-with-custom-pattern/207919/5 "2019-12-13T22:13:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.