# Grok debugger(online) works, but ELK stack doesn't

**URL:** <https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897>\
**Category:** Logstash\
**Created:** [January 26, 2017, 12:20pm UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897 "2017-01-26T12:20:56Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Joseph](https://discuss.elastic.co/u/Joseph)\
**Post date:** [January 26, 2017, 12:20pm UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/1 "2017-01-26T12:20:56Z")

</div>

I am new to logstash and grok filters. My sample logs look like this

{"level":"INFO", "date":"01/24/2017", "time":"14:22:05", "text":"[Audit]:RBAC Bundle Started"},  
{"level":"INFO", "date":"01/24/2017", "time":"14:22:09", "text":"[Audit]:Started Catalog Manager Service"},  
{"level":"WARN", "date":"01/24/2017", "time":"14:22:09", "text":"[Audit]:Got null for resourceServerBasepath"},

I am getting these logs in port 5044.

```
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

```

filter {

```
    grok {

```

match =\> { "message" =\> '"%{WORD:level}":"%{WORD:type}", "%{WORD:date}":"%{DATE\_US:date}", "%{WORD:time}":"%{TIME:timerange}", "%{WORD:text}":"%{GREEDYDATA:information}'

}  
add\_field =\> ["received\_at", "%{@timestamp}"]

}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
sniffing =\> true  
manage\_template =\> false  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}

I tried the grok debugger online and the sample data was parsed correctly. But when I employ the same in ELK stack, unparsed data gets pushed into Elasticsearch. Any idea what might be causing the trouble? Any help is appreciated.

---

<div class="post-metadata">

**Author:** ![m0nhawk](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@m0nhawk](https://discuss.elastic.co/u/m0nhawk)\
**Post date:** [January 26, 2017, 1:01pm UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/2 "2017-01-26T13:01:04Z")

</div>

It's a proper JSON log, then why not to use JSON filter for this?

Getting JSON without a trailing comma and then parsing it:

```
filter {
    grok {
        match => { "message" => "%{GREEDYDATA:json}," }
        add_field => ["received_at", "%{@timestamp}"]
    }
    json {
        source => "json"
    }
    mutate {
        remove_field => ["message"]
    }
}

```

For this I get (in `rubydebug`):

```
{
           "date" => "01/24/2017",
           "path" => "/root/logs/test.log",
     "@timestamp" => 2017-01-26T12:59:05.683Z,
          "level" => "INFO",
    "received_at" => "2017-01-26T12:59:05.683Z",
       "@version" => "1",
           "host" => "virt",
           "json" => "{\"level\":\"INFO\", \"date\":\"01/24/2017\", \"time\":\"14:22:05\", \"text\":\"[Audit]:RBAC Bundle Started\"}",
           "time" => "14:22:05",
           "text" => "[Audit]:RBAC Bundle Started"
}
```

---

<div class="post-metadata">

**Author:** ![Joseph](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Joseph](https://discuss.elastic.co/u/Joseph)\
**Post date:** [January 26, 2017, 2:57pm UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/4 "2017-01-26T14:57:27Z")

</div>

Hi. Thanks for your reply. I wasn't sure if logstash JSON filter parses the "date" field as DATE or simply as another string and so I was not sure if I could use it. Will I be able to query the "date" field in Elasticsearch with this filter?

---

<div class="post-metadata">

**Author:** ![m0nhawk](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@m0nhawk](https://discuss.elastic.co/u/m0nhawk)\
**Post date:** [January 26, 2017, 3:32pm UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/5 "2017-01-26T15:32:13Z")

</div>

Then make the following `mutate` and `date` after the `json`:

```
    mutate {
        add_field => { "datetime" => "%{date} %{time}" }
    }
    date {
        match => ["datetime", "MM/dd/yyyy HH:mm:ss"]
        target => "timestamp"
    }

```

Now you can use `timestamp` as a time field in Kibana and ElasticSearch.

```
{
          "level" => "WARN",
           "path" => "/root/logs/test.log",
       "datetime" => "01/24/2017 14:22:09",
     "@timestamp" => 2017-01-26T15:30:46.083Z,
    "received_at" => "2017-01-26T15:30:46.083Z",
       "@version" => "1",
           "host" => "virt",
           "json" => "{\"level\":\"WARN\", \"date\":\"01/24/2017\", \"time\":\"14:22:09\", \"text\":\"[Audit]:Got null for resourceServerBasepath\"}",
           "text" => "[Audit]:Got null for resourceServerBasepath",
      "timestamp" => 2017-01-24T13:22:09.000Z
}
```

---

<div class="post-metadata">

**Author:** ![Joseph](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Joseph](https://discuss.elastic.co/u/Joseph)\
**Post date:** [January 27, 2017, 9:26am UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/6 "2017-01-27T09:26:06Z")

</div>

Thanks for your inputs! They were very helpful. I am able to query the data using "timestamp". Just to make it perfect, is it possible to replace "@timestamp" with "timestamp", so that when I open kibana discover, the data is sorted according to "timestamp" and not according to "@timestamp".

---

<div class="post-metadata">

**Author:** ![m0nhawk](https://avatars.discourse-cdn.com/v4/letter/m/9de0a6/32.png) [@m0nhawk](https://discuss.elastic.co/u/m0nhawk)\
**Post date:** [January 27, 2017, 9:46am UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/7 "2017-01-27T09:46:44Z")

</div>

Yes, the `target` in `date` have a default value of `@timestamp`.

Then just delete those line and it will write the timestamp into the `@timestamp`.

---

<div class="post-metadata">

**Author:** ![Joseph](https://avatars.discourse-cdn.com/v4/letter/j/dc4da7/32.png) [@Joseph](https://discuss.elastic.co/u/Joseph)\
**Post date:** [January 27, 2017, 10:29am UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/8 "2017-01-27T10:29:20Z")

</div>

I think I am already doing that.

My filter and output look like this:

filter {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:json}," }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
}  
json {  
source =\> "json"  
}  
mutate {  
add\_field =\> { "datetime" =\> "%{date} %{time}" }  
remove\_field =\> ["message"]  
}  
date {  
match =\> ["datetime", "MM/dd/yyyy HH:mm:ss"]  
target =\> "datetime"  
}  
}

```
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "filebeat-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}

```

Though I change my target to "datetime", Kibana seems to take @timestamp as the primary time field.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 24, 2017, 10:29am UTC](https://discuss.elastic.co/t/grok-debugger-online-works-but-elk-stack-doesnt/72897/9 "2017-02-24T10:29:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.