# Grok filter does not behave as I want it to

**URL:** <https://discuss.elastic.co/t/grok-filter-does-not-behave-as-i-want-it-to/149087>\
**Category:** Logstash\
**Created:** [September 19, 2018, 8:58am UTC](https://discuss.elastic.co/t/grok-filter-does-not-behave-as-i-want-it-to/149087 "2018-09-19T08:58:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![antwan](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@antwan](https://discuss.elastic.co/u/antwan)\
**Post date:** [September 19, 2018, 8:58am UTC](https://discuss.elastic.co/t/grok-filter-does-not-behave-as-i-want-it-to/149087/1 "2018-09-19T08:58:39Z")

</div>

Hello!

I am trying to set up some filters in Logstash, so that my Elastisearch log gets parsed and pretty in Kibana.

I have exported the pipeline-plain.json regarding Elasticsearch `server` log from Filebeat, and after some tinkering with the grok filter, I ended up with:

```
   filter {
      if [fileset][module] == "elasticsearch" {
          mutate {
            add_field => {
              "raw_message" => "%{message}"
            }
          }
        if [fileset][name] == "server" {
          grok {
            pattern_definitions => {
                "GREEDYMULTILINE" => "(.|\n)*"
                "INDEXNAME" => "[a-zA-Z0-9_.-]*"
            }
            match => { "message" => ["\[%{TIMESTAMP_ISO8601:elasticsearch.server.timestamp}\]\[%{LOGLEVEL:log.level}%{SPACE}?\]\[%{DATA:elasticsearch.server.component}%{SPACE}*\](%{SPACE}*)?(\[%{DATA:elasticsearch.node.name}\])?(%{SPACE}*)?(\[gc\](\[young\]\[%{NUMBER:elasticsearch.server.gc.young.one}\]\[%{NUMBER:elasticsearch.server.gc.young.two}\]|\[%{NUMBER:elasticsearch.server.gc_overhead}\]))?%{SPACE}*((\[%{INDEXNAME:elasticsearch.index.name}\]|\[%{INDEXNAME:elasticsearch.index.name}\/%{DATA:elasticsearch.index.id}\]))?%{SPACE}*%{GREEDYMULTILINE:message}"]}
          }
          date {
            match => ["elasticsearch.server.timestamp", "ISO8601"]
          }
        }
      }
    }

```

When this filter is applied to:

```
[2018-09-17T10:45:35,501][INFO][o.e.x.s.a.s.FileRolesStore] [xTyQnIt] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]

```

I end up with:

```
"message":["[2018-09-17T10:45:35,501][INFO][o.e.x.s.a.s.FileRolesStore] [xTyQnIt] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]","parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]"],

```

This `message` is an array consisting of, from what I can understand, my `raw_message`(the whole log) and `message`(which is all I want in `message`. I am not an expert at all, but I suspect `%{GREEDYMULTILINE:message}` does something under the hood which I am not aware of..

I have done the same thing to parse Logstash logs:

Filter:

```
filter {
  if [fileset][module] == "logstash" {
      mutate {
        add_field => {
          "raw_message" => "%{message}"
        }
      }
    if [fileset][name] == "log" {
      grok {
        pattern_definitions => {
            "LOGSTASH_CLASS_MODULE" => "[\w\.]+"
            "LOGSTASH_LOGLEVEL" => "(INFO|ERROR|DEBUG|FATAL|WARN|TRACE)"
        }
        match => { "message" => ["\[%{TIMESTAMP_ISO8601:logstash.log.timestamp}\]\[%{LOGSTASH_LOGLEVEL:logstash.log.level}\s*\]\[%{LOGSTASH_CLASS_MODULE:logstash.log.module}\s*\]\s*%{GREEDYDATA:logstash.log.message}"]}
      }
      mutate {
        replace => {
          "message" => "%{logstash.log.message}"
        }
      }
      date {
        match => ["logstash.log.timestamp", "ISO8601"]
      }
    }
  }
}

```

Applied to:

```
[2018-09-05T12:39:48,285][INFO][logstash.inputs.metrics] Monitoring License OK

```

With result:

```
"message":"Monitoring License OK",

```

Since the filter for Elasticsearch does not utilize the same "json-logic", and sends data to `message` straight away, and not like log stash, `logstash.log.message` i don't do any replace-mutation.

I someone with a few minutes to spare could guide me as to what I could do, Id appreciate it immensely.

---

<div class="post-metadata">

**Author:** ![antwan](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@antwan](https://discuss.elastic.co/u/antwan)\
**Post date:** [September 19, 2018, 10:10am UTC](https://discuss.elastic.co/t/grok-filter-does-not-behave-as-i-want-it-to/149087/2 "2018-09-19T10:10:37Z")

</div>

My `raw_message` created my problem, so I sent GREEDYMULTILINE to something else that `message`, mutated this "else" into `message` and Im a happy grokker!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2018, 10:10am UTC](https://discuss.elastic.co/t/grok-filter-does-not-behave-as-i-want-it-to/149087/3 "2018-10-17T10:10:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.