# Custom grok pattern issue **URL:** \ **Category:** Logstash\ **Created:** [May 8, 2018, 5:03am UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931 "2018-05-08T05:03:14Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![AJ\_NOURI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aj_nouri/32/29741_2.png) [@AJ\_NOURI](https://discuss.elastic.co/u/AJ_NOURI)\ **Post date:** [May 8, 2018, 5:03am UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931/1 "2018-05-08T05:03:15Z")
I am testing custom grok patterns. First with the following logstash config: ``` input { stdin{} } filter { grok { match => {"message" => "%{SYSLOGTIMESTAMP:time} %{GREEDYDATA:other}"} } } output { stdout { codec => rubydebug } } ``` When I submit `May 8 06:47:27 aef46fa42c11[1036]: 29.22.234.151` The result is as expected ``` { "@version" => "1", "message" => "May 8 06:47:27 aef46fa42c11[1036]: 29.22.234.151", "host" => "scw-8ccfeb", "@timestamp" => 2018-05-08T04:52:55.115Z, "time" => "May 8 06:47:27", "other" => "aef46fa42c11[1036]: 29.22.234.151" } ``` But when I try to include in the match the custom pattern, it doesn't work: `match => {"message" => "%{SYSLOGTIMESTAMP:time} (?[a-z0-9]+\[[0-9]+\]) %{GREEDYDATA:other}"}` The same input gives grokparsefailure: `May 8 06:47:27 aef46fa42c11[1036]: 29.22.234.151` result: ``` { "host" => "scw-8ccfeb", "@version" => "1", "tags" => [ [0] "_grokparsefailure" ], "message" => "May 8 06:47:27 aef46fa42c11[1036]: 29.22.234.151", "@timestamp" => 2018-05-08T04:58:50.598Z } ``` I have tested the regular expression ([http://rubular.com/](http://rubular.com/) as well as [https://regex101.com/](https://regex101.com/)) and it looks good: ![Selection_001_08_05](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a3ca2a437a1533bbeb8ff01dc8d61935e274f7ec.jpg) I doubt, the way I implemented it in match expression. Any hint is appreciated. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 8, 2018, 6:03am UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931/2 "2018-05-08T06:03:10Z")
Always format Logstash configuration as preformatted text so it doesn't get mangled. The problem is that you're not taking the colon after the program/pid into account. ---
**Author:** ![AJ\_NOURI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aj_nouri/32/29741_2.png) [@AJ\_NOURI](https://discuss.elastic.co/u/AJ_NOURI)\ **Post date:** [May 8, 2018, 6:17pm UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931/3 "2018-05-08T18:17:51Z")
Thanks @magnusbaeck for your reply. Indeed, if I include the colon ":" in the expression, I got no grok failure, but I get sourceid field with the colon: ``` May 8 04:31:02 aef46fa42c11[1036]: 170.66.228.69 - - { "time" => "May 8 04:31:02", "clientip" => "170.66.228.69", "message" => "May 8 04:31:02 aef46fa42c11[1036]: 170.66.228.69 - - ", "other" => "- - ", "@version" => "1", "@timestamp" => 2018-05-08T14:37:29.054Z, "sourceid" => "aef46fa42c11[1036]:", "host" => "scw-4bed5f" } ``` I have tried to substitute ":" with "", but doesn't work. ``` match => {"message" => "%{SYSLOGTIMESTAMP:time} (?[a-z0-9]+\[[0-9]+\]\:) %{IP:clientip} %{GREEDYDATA:other}"} } mutate { gsub => [ "sourceid", ':',"" ] } ``` Result is the same: ``` { "time" => "May 8 04:31:02", "clientip" => "170.66.228.69", "message" => "May 8 04:31:02 aef46fa42c11[1036]: 170.66.228.69 - - ", "other" => "- - ", "@version" => "1", "@timestamp" => 2018-05-08T14:37:29.054Z, "sourceid" => "aef46fa42c11[1036]:", "host" => "scw-4bed5f" } ``` Can I indicate it in the grok match so it ignore it? I tried adding the colon just after the custom patten, but it gives grokfailure: ``` match => {"message" => "%{SYSLOGTIMESTAMP:time} (?[a-z0-9]+\[[0-9]+\]): %{IP:clientip} %{GREEDYDATA:other}"} } ``` ---
**Author:** ![AJ\_NOURI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aj_nouri/32/29741_2.png) [@AJ\_NOURI](https://discuss.elastic.co/u/AJ_NOURI)\ **Post date:** [May 8, 2018, 6:22pm UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931/4 "2018-05-08T18:22:32Z")
Got it by escaping the colon outside of the pattern! ``` filter { grok { match => {"message" => "%{SYSLOGTIMESTAMP:time} (?[a-z0-9]+\[[0-9]+\])\: %{IP:clientip} %{GREEDYDATA:other}"} } } ``` Correct result: ``` May 8 04:31:02 aef46fa42c11[1036]: 170.66.228.69 - - { "time" => "May 8 04:31:02", "other" => "- -", "clientip" => "170.66.228.69", "host" => "scw-4bed5f", "@version" => "1", "@timestamp" => 2018-05-08T18:21:08.877Z, "message" => "May 8 04:31:02 aef46fa42c11[1036]: 170.66.228.69 - -", "sourceid" => "aef46fa42c11[1036]" } ``` Thanks @magnusbaeck! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 8, 2018, 7:21pm UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931/5 "2018-05-08T19:21:02Z")
The colon obviously needs to go outside the parenthesis group to not get captured to the field, but there's no point in escaping the colon. Colons have no special meaning in regexps. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 5, 2018, 7:21pm UTC](https://discuss.elastic.co/t/custom-grok-pattern-issue/130931/6 "2018-06-05T19:21:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.