# Grok Filter Parsing Failure **URL:** \ **Category:** Logstash\ **Created:** [February 22, 2017, 10:10am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013 "2017-02-22T10:10:47Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![agrawalsaurabh](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Post date:** [February 22, 2017, 10:10am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/1 "2017-02-22T10:10:47Z")
Hi, I was able to integrate Filebeat/logstash/Elasticsearch and kibana and able to parise logs using default APACHE LOG filter. Next I tried to parse specific information and push it to Elasticsearch. However, logstash console shows me a grok filter failure message: My grok pattern in logstash.conf file is: > filter { > grok { > match =\> { "message" =\> "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"} > } > } The exception thrown is as follows: > { > "@timestamp" =\> 2017-02-22T09:51:22.926Z, > "offset" =\> 2995, > "@version" =\> "1", > "input\_type" =\> "log", > "beat" =\> { > "hostname" =\> "LT0004658", > "name" =\> "LT0004658", > "version" =\> "5.2.0" > }, > "host" =\> "LT0004658", > "source" =\> "D:\Softwares\LogMonitoring\logs\access-161212.log", > "message" =\> "10.129.34.54 10.129.1.180 - - [12/Dec/2016:12:55:56 +0530] > "GET /\_ui/desktop/theme-marketplace/img/fulfilled.png HTTP/1.1" 200 1824 "htt > ps://uat-jiomall.ril.com/search/?search-category=All&keyword=shirt" "Mozilla/5 > .0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.28 > 40.99 Safari/537.36" "image/png" 200 4189 [Response:0]", > "type" =\> "log", > "tags" =\> [ > [0] "beats\_input\_codec\_plain\_applied", > [1] "\_grokparsefailure" > ] > } Any help in this regard is much appreciated. Thanks. Saurabh ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 22, 2017, 10:28am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/2 "2017-02-22T10:28:23Z")
Why not start with the standard Apache Combined pattern instead of building everything from scratch? Your current log entry looks like a standard Apache Combined line with an extra IP address in the beginning and a couple of extra fields at the end. ---
**Author:** ![agrawalsaurabh](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Post date:** [February 22, 2017, 10:30am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/3 "2017-02-22T10:30:54Z")
So you mean to say use the following format and it will parse everything for me ? > input { > beats { > port =\> 5044 > } > } > filter { > grok { > match =\> { "message" =\> "%{COMBINEDAPACHELOG}"} > } > } > output { > elasticsearch { > hosts =\> "localhost:9200" > #manage\_template =\> false > #index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" > #document\_type =\> "%{[@metadata][type]}" > } > > stdout { codec =\> rubydebug } > } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 22, 2017, 10:39am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/4 "2017-02-22T10:39:15Z")
No, but you can use COMBINEDAPACHELOG as a starting point for your expression. Start by changing the pattern to `%{IP:whatever} %{COMBINEDAPACHELOG}` to pick up the extra IP address. Does that work so far? Then continue with the extra fields at the end of the line. ---
**Author:** ![agrawalsaurabh](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Post date:** [February 23, 2017, 9:36am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/5 "2017-02-23T09:36:24Z")
How do we use key/value pairs in the match format ? For instance: %{IP:xyz}, in this is IP a predefined value which needs to be used? Similarly what value will be "xyz" in this example? Let's say in apache log format, I use a custom log format %T which prints the time taken to serve the request. Now I want to parse this in logstash, how will I do it is the question? Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 23, 2017, 10:23am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/6 "2017-02-23T10:23:50Z")
> For instance: %{IP:xyz}, in this is IP a predefined value which needs to be used? IP is a grok pattern. They are defined here: [https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns](https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns) > Similarly what value will be "xyz" in this example? That's the name of the field where the string matched by the pattern will be placed. > Let's say in apache log format, I use a custom log format %T which prints the time taken to serve the request. Now I want to parse this in logstash, how will I do it is the question? %T expands to a floating point number, right? Then NUMBER would be a suitable pattern: ``` %{NUMBER:duration} ``` Additionally, to make sure the field is stored as a floating point number (rather than a string): ``` %{NUMBER:duration:float} ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 23, 2017, 10:23am UTC](https://discuss.elastic.co/t/grok-filter-parsing-failure/76013/7 "2017-03-23T10:23:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.