Hi All,
I would like to ask you for help. I am trying to parse Microsoft DNS debug logs but I stucked.
My testing environment is running on ELK version 7.2. Beats are also running on 7.2 version.
Problem is I am getting _grokparsefailure also in case that grok debuger show me it should work.
My config files
input.conf:
input {
beats {
port => 5051
type => "dns"
}
}
dns.conf:
filter {
if [type] == "dns" {
if [message] =~ /^$/ {
drop { }
} else {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => { "Message" => "%{MS_DNS_DATE:date}\s+%{TIME:time}\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{DATA:response}\s+%{WORD:dns_query_type}\s+[%{BASE16NUM:hex_flags}\s+%{WORD:flags}\s+%{WORD:rcode_name}]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}"}
match => { "Message" => "%{MS_DNS_DATE:date}\s+%{TIME:time}\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{DATA:response}\s+%{WORD:dns_query_type}\s+[%{BASE16NUM:hex_flags}\s+%{WORD:flags}\s+%{WORD:recursion}\s+%{WORD:rcode_name}]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}"}
match => { "Message" => "%{MS_DNS_DATE:date}\s+%{TIME:time}\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{DATA:response}\s+%{WORD:dns_query_type}\s+[%{BASE16NUM:hex_flags}\s+%{WORD:recursion}\s+%{WORD:rcode_name}]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}"}
match => { "Message" => "%{MS_DNS_DATE:date}\s+%{TIME:time}\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{DATA:response}\s+%{WORD:dns_query_type}\s+[%{BASE16NUM:hex_flags}\s+%{WORD:rcode_name}]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}"}
match => { "Message" => "%{MS_DNS_DATE:date}\s+%{TIME:time}\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{WORD:dns_query_type}\s+[%{BASE16NUM:hex_flags}\s+%{WORD:flags}\s+%{WORD:rcode_name}]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}"}
}
}
}
}
custom pattern:
MS_DNS_DATE %{MONTHDAY}. %{MONTHNUM}. %{YEAR}
Sample log line: (date is with spaces so i made custom patern)
28. 10. 2019 10:52:48 0A38 PACKET 000000DE7BBC74B0 UDP Snd 192.168.5.201 98af Q [0000 NOERROR] SOA (4)mell(2)cz(0)
If i try to parse that sample in GROK DEBUGGER it works with pattern:
%{MS_DNS_DATE:date}\s+%{TIME:time}\s+%{DATA:thread_id}\s+%{WORD:dns_type}\s+%{BASE16NUM:packet_id}\s+%{WORD:dns_protocol}\s+%{WORD:dns_direction}\s+%{IP:dns_ip}\s+%{BASE16NUM:xid}\s+%{DATA:response}\s+%{WORD:dns_query_type}\s+[%{BASE16NUM:hex_flags}\s+%{WORD:rcode_name}]\s+%{WORD:query_type_name}\s+%{GREEDYDATA:dns_domain}
Can someone help me with debugging?
Thanks.
Jan