BIND DNS Pattern

Elastic Stack Logstash
1

Hi All,
I am very new to logstash and hope I could get some help regarding splitting a log file from BIND DNS server to be pushed to Elastic Search.

The log files have the following format:
04-Nov-2015 08:28:39.261 queries: info: client 192.168.169.122#59319: query: istatic.eshopcomp.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.269 queries: info: client 192.168.212.136#48872: query: idsync.rlcdn.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.269 queries: info: client 192.168.19.61#53970: query: 3-courier.sandbox.push.apple.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.270 queries: info: client 192.168.169.122#59319: query: ajax.googleapis.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.272 queries: info: client 192.168.251.24#37028: query: um.simpli.fi IN A + (10.10.80.50) 04-Nov-2015 08:28:39.272 queries: info: client 192.168.251.24#37028: query: www.wtp101.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.273 queries: info: client 192.168.251.24#37028: query: magnetic.t.domdex.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.273 queries: info: client 172.25.111.175#59612: query: api.smoot.apple.com IN A + (10.10.80.50) 04-Nov-2015 08:28:39.275 queries: info: client 192.168.7.181#45913: query: www.miniclip.com IN A + (10.10.80.50)

The following configuration I have tried however I am not able to split the different information into different fields:
`input {
file {
type => "speedtest"
path => [ "/data/bind" ]
start_position => "beginning"
}
}

filter {
grok {
match => ["messages", "%{URIHOST}:40%{ISO8601_TIMEZONE} %{WORD:queries} %{WORD:info} %{WORD:client} %{IP:client} %{WORD:query} %{URIPATHPARAM:request} %{WORD:IN} %{WORD:DNSQuery} + %{IP:DNSServer}"]
}
mutate {
split => ["messages", " "]
}
kv{
field_split => " "
}
}

output {
elasticsearch {
protocol => "node"
host => "localhost"
cluster => "elasticsearch"
}
}
`

In Kibana all jus fall under one source field item not split:
{ "_index": "logstash-2015.11.30", "_type": "speedtest", "_id": "AVFXlcqj54iGEHVC1dgQ", "_score": null, "_source": { "message": "09-Nov-2015 14:15:07.879 queries: info: client 192.168.242.98#34081: query: fbcdn-photos-a-a.akamaihd.net IN A + (10.10.96.50)", "@version": "1", "@timestamp": "2015-11-30T08:50:13.218Z", "host": "localhost.localdomain", "path": "/data/bind", "type": "speedtest", "tags": [ "_grokparsefailure" ] }, "fields": { "@timestamp": [ 1448873413218 ] }, "sort": [ 1448873413218 ] }

Also the time is taken when it was uploaded not the first field that is the date and the second one the time. Any assistance would be truly appreciated.

2

There are multiple problems preventing this from working.

  • The name of the field is message but your filters are trying to work against the non-existent messages field.
  • The grok expression isn't even close to matching the log entries.
  • Why are you splitting the message on spaces?
  • Why the kv filter? I don't see any key/value data anywhere.

Suggestions:

  • To speed up your cycle time, disable the elasticsearch output and just use stdout { codec => rubydebug } until things start to look good.
  • To avoid having to delete the sincedb file for each iteration, either set sincedb_path to /dev/null or use the stdin input instead of the current file input to feed Logstash logs via stdin.
  • Take some time to study regular expressions, then use http://grokconstructor.appspot.com/ as help to construct a grok expression that matches the logs.
3

Hi Magnus,

Thank you very much for the pointers. I have been able to almost get it correct however I am struggling with getting the last + and () around the IP address resolved. Here is the latest filer:

%{GREEDYDATA:timestamp} %{HAPROXYTIME} queries: info: client %{IPORHOST:clientip}#%{NUMBER:port}: query: (?[a-z0-9-]+.[a-z0-9-]+\S+) IN %{WORD:recType} + (%{IPORHOST:DNSip})

Do you have any suggestion on how to get the Matched : + (10.10.80.50)

Your assistance would be appreciated.

4

I strongly suggest that you don't use GREEDYDATA to match the timestamp. Something like

(?<timestamp>%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME})

should work (assuming that all month names are in English).

As for the plus signs and parentheses, keep in mind that those characters have special meaning in regular expressions. To match them literally you need to escape them with a backslash.

5

Thank you Magnus, I have now also fixed the last part with the slashes. Now when I use the online tools they all show that the Grok statement is correct now however when I run it through the filer I get " _grokparsefailure " could is it maybe possible to spot the challange why the fields are not identified here is the filter:

input {
file {
type => "BIND_DNS"
path => [ "/data/bind" ]
start_position => "beginning"
}
}

filter {
grok {
match => ["messages","(?%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}) queries: info: client %{IPORHOST:clientip}#%{NUMBER:port}: query: (?[a-z0-9-]+.[a-z0-9-]+\S+) IN %{WORD:recType} + (%{IPORHOST:DNSIP})"]
}
}

output {
elasticsearch {
protocol => "node"
host => "localhost"
cluster => "elasticsearch"
}
}

6

As I said in my very first response, the name of the field is message but your filters are trying to work against the non-existent messages field.

7

Sorry now I understand what you were trying to say with that :slight_smile:

8

Thank you very much Magnus all is working now, much appreciated.

9

Would you please chare your working config as i am trying to do the same here ... would really appreciate ..
thanks
Maile,.