Grok parsing in logstash for Windows DNS Debug Log

Elastic Stack Logstash
1

I'm using grok in Logstash (7.8.0) to parse data from a Windows Server (2019) DNS debug log (sent via filebeat) using the statement below. Most of the time, the data gets parsed correctly and fields are populated and visible in Kibana. However, roughly 1/3 of the time, the data doesn't appear to get parsed and the fields aren't visible in Kibana. In this scenario, I know the data is getting from Logstash to Elasticsearch because the message field in Kibana is populated with the entry from the DNS debug log. I've confirmed that the pattern is the same between data that does and doesn't correctly get parsed. In some cases, there may be 10-20 entries with the same timestamp (to the second) in the DNS log. Is this potentially a situation where Logstash / grok just can't keep up?

    grok {
      match => {
        "message" => [
		"(?<timestamp>%{DATE_US} %{TIME} (?:AM|PM))\s+%{NUMBER}\s+%{WORD:t1}\s+%{BASE16NUM}\s+%{WORD:network.transport}\s+%{WORD:network.direction}\s+%{IP:dns.resolved_ip}\s+%{BASE16NUM}\s+%{WORD:dns.op_code}\s+\[%{BASE16NUM}\s+%{WORD:dns.header_flags}\s+%{WORD:dns.response_code}\]\s+%{WORD:dns.question.type}\s+%{GREEDYDATA:dns.question.name}"
        ]
      }
    }
2

I very much doubt it. Do the unparsed events have a _grokparsefailure tag?

Maybe WORD doesn't match your data. WORD is \b\w+\b and \w is [a-zA-Z0-9_] so it accepts underscore but not hyphen or period.

3

Hi Badger. Would the _grokparsefailure be visible in Kibana? If so, it's not there for the events in question. Below is a sample log entry. All fields with the WORD designation are text. Thanks

12/6/2021 2:54:31 PM 2518 PACKET 00000230C7FC09D0 UDP Rcv 192.168.1.1 2be4 Q [0001 D NOERROR] A (7)outlook(2)ha(9)office365(3)com(0)

4

The tags field should certainly be visible in kibana. That grok pattern matchs that message for me, so I have no idea what could be going wrong.

5

Yea, it seems to skip the parsing process for certain lines all together, as the fields (dns.resolved_ip, etc.) don't even exist in Kibana for those entries. Also, only the dns value exists in the tags field, which gets applied from filebeat.

6

Is it possible that one of your beats is sending directly to Elasticsearch and bypassing logstash?

7

No, everything is going through Logstash. There's another field, outside the grok, that gets added in the Logstash config. When the issue occurs, that field exists in Kibana, just not any fields from the grok statement.

8

I would start looking at conditionals and routing. Does the grok always get executed? Are you routing events beween pipelines?

9

Ok thanks, I'll do some more digging. The grok gets executed anytime "dns" is in the tags field (assigned by filebeat). I checked the bad entries in Kibana, and the dns tag is there. I do use a single filebeat instance to monitor two different log locations (dhcp and dns) on the server, but didn't think that'd be an issue. The dhcp and dns data ultimately end up in the same index in Elastic.

10

Do the events have a _grokparsefailure tag in the tags field? If not then either the grok filter did not execute, or the [message] field does not exist.

11

I do see the _grokparsefailure tag on the bad entries now. I didn't initially realize, but there was a remove_tag entry removing it at the end of the config.

12

I had the field after the timestamp set to NUMBER, but it occasionally included a letter, so I changed it to BASE16NUM. The WORD:dns.header_flags field is also occasionally blank, so I switched it to GREEDYDATA:dns.header_flags. I haven't seen an unparsed entry since. Thanks for your help!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.