Extracting from Bind 9 log files

Elastic Stack Logstash
1

So I've been basically testing and experimenting with the whole ELK and i'm impressed with its versatility and already have a few things being displayed from various data sources which pleases my boss and my team quite nicely, to the point where i get coffee made for me in the mornings LOL

I've now got something I'm just not able to get around and I've been trying for about a week to make it work.

message:client, 127.0.0.1#56692, (microsoft.com):, query:, microsoft.com, IN, A, +, (127.0.0.1) @version:1 @timestamp:February 20th 2017, 12:46:47.533 path:/var/log/bind9/query.log host:ip-127.0.0.1 tags:_grokparsefailure id:AVpbj2TWl5ZobaQBDUM _type:logs _index:logstash-2017.02.20 _score:1

I am trying to extract as a base metric either (microsoft.com) or microsoft.com to i can see the most queried domains on a DNS server. I've tried basic file filters, which show the data but do not allow me to query the 'message' field beyond showing the full field data.

I've tried reg ex, but i get inconsistent results, and i think thats due to the amount of time its taking to run on a larger log file ( note to self, must try a smaller log file to check that ) using /^(query?://)?([\da-z.-]+).([a-z.]{2,6})([/\w .-])/?$/ but i must admit, my developer who gave me this was somewhat distracted by pokemon go at the time.

I've tried building the query using grok to parse out the fields so i can at least see what i need to see ...

 grok {
match => ["message", "%{WORD:query:,}  %{WORD:query} %{WORD:IN}"]
}

But this doesn't seem to drop new fields into my kibana discovery console.

I peeled back to basics and started on a copy to the Apache access log i had and tried to build up on that, but i think i might be running in circles as this pretty much does the same thing as my existing conf file below

input {
file {
path => "/var/log/bind9/query.log"
start_position => beginning
}
}

filter {
grok {
match => ["message", "%{WORD:query:,} %{WORD:query} %{WORD:IN}"]
}
mutate {
split => ["message", " "]
}
kv{
field_split => " "
}
}

output {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}

Has anyone else ever had to extract a single bit of datum from the message field before that doesn't have any identifiable markers to work from ?

2

so narrowing it down by using this
\s*([-a-z-].[a-z]):\s*

i get microsoft.com, IN, A, +, (127.0.0.1)... still working on removing or screening out the IN, A, +, (127.0.0.1) part

3

I'd use a bunch of %{NOTSPACE}, cleaner than the regexp.

4

tried a bunch of them last night including trying to mutate the output of my code, im not an expert by any imagination, but i know im doing something wrong, each time i read the grok parsing instructions and the grok patterns page ( https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns ) i just slid further and further into confusion.

I think i have to use a drop filter, but everything im reading says drop filters drop whole lines so im not sure... otherwise, i need to find a way to insert the results into a string and then query the string somehow.
i can't believe no ones come across the need to extract and report on a single piece of datum before and not hit this wall.

5

by the way

http://grokconstructor.appspot.com FTW!!!!

ive gained a better understanding of grok patterns from testing with this and reading endless blogs and posts that only seem to be concerned with numbers and apache logs

1 Like
6

\A%{NOTSPACE}%{SPACE}%{NOTSPACE}%{JAVALOGMESSAGE} gives me the same output as \s*([-a-z-].[a-z]):\s*

im definitely running in circles now

7

ok, i need some help i just can't figure out what the correct pattern would be can anyone lend a hand ?

8

finally managed to figure this out

input {
file {
path => "/var/log/bind9/query.log"
start_position => beginning
}
}

filter {
grok {
match => {"message" => "client %{IP:clientip}#%{POSINT:clientport} (%{GREEDYDATA:query}): query: %{GREEDYDATA:Target} IN %{GREEDYDATA:querytype} (%{IP:dns})"}
}
}

output {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}

this is my conf file for extracting the DNS names of DNS queries into Kibana, it works, so now im mining data :slight_smile:

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.