DNS Queries grok pattern working on devtools but not in kibana dashboards

[Poubelle_Dirty]

Hello,
I'm new to ELK stack and I encounter a problem.
I'm using the DNS grok pattern from here to parse dns queries from my bind server : https://github.com/cjslack/grok-debugger/blob/master/public/patterns/bind

It works perfectly to parse those kind of messages :

26-Aug-2023 16:22:11.872 queries: info: client @0x7fc8302cb070 192.168.1.75#15805 (discuss.elastic.co): query: discuss.elastic.co IN AAAA +T (192.168.1.57)

But in the logs, I can see that I also have those messages with HTTPS type and the native grok filter from github doesn't include this "HTTPS" Type :

26-Aug-2023 16:32:13.096 queries: info: client @0x7fc83023aff0 192.168.1.75#16181 (consent.cookiebot.eu): query: consent.cookiebot.eu IN HTTPS +T (192.168.1.57)

So I added "HTTPS" in the grok pattern like this :

BIND9_DNSTYPE (?:A|AAAA|HTTPS|CAA|CDNSKEY|CDS|CERT|CNAME|CSYNC|DLV|DNAME|DNSKEY|DS|HINFO|LOC|MX|NAPTR|NS|NSEC|NSEC3|OPENPGPKEY|PTR|RRSIG|RP|SIG|SMIMEA|SOA|SRV|TSIG|TXT|URI)

When trying this updated pattern in the kibana devtools with the grok debugger, it parse the logs correctly but not in my kibana dashboard, where I get a grokparsefailure each time the dns query type is "HTTPS" (other type like A and AAAA) are working perfectly.

I tried to reboot my entire stack and machine but the problem is not there.
Thank you !

[Badger]

If I override all the patterns in the grok filter itself

    grok {
        pattern_definitions => {
"BIND9_TIMESTAMP" => "%{MONTHDAY}[-]%{MONTH}[-]%{YEAR} %{TIME}"
"BIND9_DNSTYPE" => "(?:A|AAAA|CAA|CDNSKEY|CDS|CERT|CNAME|CSYNC|DLV|DNAME|DNSKEY|DS|HINFO|LOC|MX|NAPTR|NS|NSEC|NSEC3|OPENPGPKEY|PTR|RRSIG|RP|SIG|SMIMEA|SOA|SRV|TSIG|TXT|URI)"
"BIND9_CATEGORY" => "(?:queries)"
"BIND9_QUERYLOGBASE" => "client(:? @0x(?:[0-9A-Fa-f]+))? %{IP:[client][ip]}#%{POSINT:[client][port]:int} \(%{GREEDYDATA:[bind][log][question][name]}\): query: %{GREEDYDATA:[dns][question][name]} (?<[dns][question][class]>IN) %{BIND9_DNSTYPE:[dns][question][type]}(:? %{DATA:[bind][log][question][flags]})? \(%{IP:[server][ip]}\)"

"BIND9_QUERYLOG" => "%{BIND9_TIMESTAMP:timestamp} %{BIND9_CATEGORY:[bind][log][category]}: %{LOGLEVEL:[log][level]}: %{BIND9_QUERYLOGBASE}"
        }
        match => { "message" => "%{BIND9_QUERYLOG}" }
    }

Then it parses the regular log entry just fine. If I change|HINFO|LOC| to |HINFO|HTTPS|LOC| then it also parses the HTTPS log entry.

So the question is where exactly are you modifying the BIND9_DNSTYPE pattern and why do you expect the grok filter to use the modified pattern?

[Poubelle_Dirty]

Thank you for helping. In my logstash configuration file, I use this :

filter {
  grok {
    patterns_dir => ["./patterns"]
    match => { "message" => "%{BIND9-jij}" }
  }

And in the "patterns" dir, I have the pattern definition within a file named "bind" :

BIND9_TIMESTAMP %{MONTHDAY}[-]%{MONTH}[-]%{YEAR} %{TIME}

BIND9_DNSTYPE (?:A|AAAA|CAA|CDNSKEY|CDS|CERT|CNAME|CSYNC|DLV|DNAME|DNSKEY|DS|HINFO|HTTPS|LOC|MX|NAPTR|NS|NSEC|NSEC3|OPENPGPKEY|PTR|RRSIG|RP|SIG|SMIMEA|SOA|SRV|TSIG|TXT|URI)
BIND9_CATEGORY (?:queries)

# dns.question.class is static - only 'IN' is supported by Bind9
# bind.log.question.name is expected to be a 'duplicate' (same as the dns.question.name capture)
BIND9_QUERYLOGBASE client(:? @0x(?:[0-9A-Fa-f]+))? %{IP:[client][ip]}#%{POSINT:[client][port]:int} \(%{GREEDYDATA:[bind][log][question][name]}\): query: %{GREEDYDATA:[dns][qu
estion][name]} (?<[dns][question][class]>IN) %{BIND9_DNSTYPE:[dns][question][type]}(:? %{DATA:[bind][log][question][flags]})? \(%{IP:[server][ip]}\)

# for query-logging category and severity are always fixed as "queries: info: "
BIND9_QUERYLOG %{BIND9_TIMESTAMP:timestamp} %{BIND9_CATEGORY:[bind][log][category]}: %{LOGLEVEL:[log][level]}: %{BIND9_QUERYLOGBASE}

BIND9-jij %{BIND9_QUERYLOG}

I Just tried the "pattern_definitions" as you did and it works perfectly for https so it's seems that my patterns_dir is not working

It's really weird because I changed on purpose the name BIND9 by BIND9-jij to see if logstash is loading the pattern. When i spell it wrong in my logsyash conf file (for example (BIND9-jijiji), it obviously fails with an errror saying it can't load the pattern file but here when spelled correctly (BIND9-jij), there is no error so it seems everything is loaded perfectly....

[Badger]

logstash bundles a different set of BIND9 patterns.

# cat /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-patterns-core-4.3.4/patterns/legacy/bind
BIND9_TIMESTAMP %{MONTHDAY}[-]%{MONTH}[-]%{YEAR} %{TIME}
 
BIND9 %{BIND9_TIMESTAMP:timestamp} queries: %{LOGLEVEL:loglevel}: client(:? @0x(?:[0-9A-Fa-f]+))? %{IP:clientip}#%{POSINT:clientport} \(%{GREEDYDATA:query}\): query: %{GREEDYDATA:query} IN %{GREEDYDATA:querytype} \(%{IP:dns}\)

so there is always going to be BIND9 pattern defined. Try changing your grok match to [message]" => "%{BIND9_QUERYLOG}" and I suspect that will be an undefined pattern.

The issue may be patterns_dir => ["./patterns"]. That is relative to the current working directory of the logstash instance, and, particularly if you are running as a service, that may not be what you expect. Can you try an absolute path there?

[Poubelle_Dirty]

Hello,

Thank you for helping, it was indeed the problem. With a full path as "pattern_dirs", it now works.

Have a nice day !