# GROK pattern issue **URL:** \ **Category:** Logstash\ **Created:** [January 17, 2023, 2:10pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330 "2023-01-17T14:10:23Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [January 17, 2023, 2:10pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330/1 "2023-01-17T14:10:23Z")
Hello, I am using filebeat to send logs from a DNS server to logstash before being ingested in Elasticsearch. The same pattern is repeated with two spaces and one space due to the variation in logs over updates which is generally reverted. **None of the logs have \_grokpatternfailure** However, I see that the actions field is capturing the term `gravity` instead of the actual action of either `cached`, `blocked`, `query`, `reply`, `others`. The term GRAVITY is taken in case the logs have `blocked` action; I am unsure why. Can someone help me with the GROK pattern changes below? Few logs of `blocked` entries ```auto Jan 17 19:35:57 dnsmasq[879]: gravity blocked browser.pipe.aria.microsoft.com is 0.0.0.0 Jan 17 19:35:57 dnsmasq[951]: gravity blocked browser.pipe.aria.microsoft.com is NODATA Jan 17 19:35:58 dnsmasq[951]: gravity blocked app-measurement.com is 0.0.0.0 ``` Following is the GROK pattern: ```auto filter { grok { patterns_dir => ["/etc/logstash/patterns/"] match=> ["message", "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: gravity %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:action}\[DNSKEY] %{DATA:query_hostname} to 127.0.0.1", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} result is %{WORD:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV6:resolved_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} to 127.0.0.1", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:resolved_ip} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:resolved_ip} is %{GREEDYDATA:query_hostname}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{DATA:query_hostname} from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{IPV4:query_arpa}.in-addr.arpa from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:query_arpa}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{DATA:query_hostname} from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} ", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:query_type}\[DS] %{DATA:query_hostname} to 127.0.0.1", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:query} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:list} %{DATA:rDNS} is %{GREEDYDATA:local_hostname}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{IPV4:query_ptr}.in-addr.arpa from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: gravity %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:action}\[DNSKEY] %{DATA:query_hostname} to 127.0.0.1", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} result is %{WORD:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV6:resolved_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} to 127.0.0.1", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:resolved_ip} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:resolved_ip} is %{GREEDYDATA:query_hostname}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{DATA:query_hostname} from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{IPV4:query_arpa}.in-addr.arpa from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:query_arpa}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{DATA:query_hostname} from %{IPV4:src_ip}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} ", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:query_type}\[DS] %{DATA:query_hostname} to 127.0.0.1", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:query} is %{GREEDYDATA:result}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:list} %{DATA:rDNS} is %{GREEDYDATA:local_hostname}", "message","%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action}\[%{WORD:query_type}] %{IPV4:query_ptr}.in-addr.arpa from %{IPV4:src_ip}"] } ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 17, 2023, 2:16pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330/2 "2023-01-17T14:16:24Z")
> [@parthmaniar](#): > > `patterns_dir => ["/etc/logstash/patterns/"]` Please share the patterns you have in this path so people can try to replicate your issue. Also, share the output you are getting and the expected output. ---
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [January 17, 2023, 2:29pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330/3 "2023-01-17T14:29:20Z")
Hello I sincerely aplogise that I forgot to add the patterns ```auto logdate [\w]{3}\s[\s\d]{2}\s\d\d\:\d\d\:\d\d blocklist [\/\w\.]+ domain [\w\.\-]+ clientip \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} ip \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} ``` Also, the expected output is to take action `blocked` and index it to the field `action`. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/7/4707571d6d20f4bfa20d1427dcf0602ca54bfc35.png) Here the value gravity should be blocked. Here are a few logs from other fields **action == reply** `Jan 17 19:56:57 dnsmasq[879]: reply d10zdin3c2kqf3.cloudfront.net is 18.66.53.110 ` **action == query** `Jan 17 19:56:57 dnsmasq[879]: query[A] d10zdin3c2kqf3.cloudfront.net from 10.0.0.10` **action == cached** `Jan 17 19:56:57 dnsmasq[951]: cached web.facebook.com is ` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 17, 2023, 3:48pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330/4 "2023-01-17T15:48:08Z")
> [@parthmaniar](#): > > Following is the GROK pattern: First, the match option to a grok filter is a hash. logstash will convert an array to a hash pairwise, but I find it clearer to use a hash in the configuration file. Second, I would recommend matching [message] against an array of patterns rather than using multiple entries in the `match` hash. So I would use ``` match=> { "message" => [ "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: gravity %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{DATA:action}\[DNSKEY] %{DATA:query_hostname} to 127.0.0.1", .... ] } ``` Third, anchor your patterns. grok is expensive when it fails to match, not when it matches. If your lines start with the timestamp then add ^ at the start of each pattern. ``` match=> { "message" => [ "^%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", .... ``` Fourth, you are matching the first part of the line over and over and over again, which is expensive. It will be cheaper to split this into two groks ``` grok { match=> { "message" => ["^%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{GREEDYDATA:[@metadata][restOfLine]}" } } grok { match=> { "[@metadata][restOfLine]" => [ "^%{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "^gravity %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "^%{DATA:action}\[DNSKEY] %{DATA:query_hostname} to 127.0.0.1", .... ] } ``` Fifth, instead of repeating the patterns with either one or two spaces, use a regexp that matches both ``` "%{MONTH:date}\s{1,2}%{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: ... ``` Sixth, each of the patterns in the array is matched in order. The first one wins ``` "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: gravity %{WORD:action} %{DATA:query_hostname} is %{IPV4:resolved_ip}", ``` `%{WORD:action}` will match gravity, and then `%{DATA:query_hostname}` will match everything up to `is`. The second, more specific pattern, should come before the first. Seventh, there is no way to distinguish between these two patterns. The first will always match anything that would match the second ``` "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:resolved_ip} is %{GREEDYDATA:result}", "%{MONTH:date} %{MONTHDAY:date} %{TIME:date} %{WORD:service}\[%{NUMBER:pid}\]: %{WORD:action} %{IPV4:resolved_ip} is %{GREEDYDATA:query_hostname}", ``` Eighth, you may be able to disambiguate some of your patterns by replacing DATA or GREEDYDATA with more specific patterns like NOTSPACE or HOSTNAME. ---
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [January 17, 2023, 4:00pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330/5 "2023-01-17T16:00:54Z")
Thank you very much. I will work on your recommendations (although I expect them to take time given the depth and quality of your recommendation [touching CPU cycles :D]). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 14, 2023, 4:01pm UTC](https://discuss.elastic.co/t/grok-pattern-issue/323330/6 "2023-02-14T16:01:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.