I'm trying to figure out the best way to count total DNS Queries per hour or day. I can easily adjust period. But what would be the best way to count this?
Packetbeat is sending data to elasticsearch and i use kibana to visualize.
So far I've created a visualization like this:
Metrics
Y-Axis Unique count of _id
Buckets
X-Axis @timestamp per hour
At first it did the trick and then started displaying errors like "10 of 21 shards failed" with " No results found".
I could also use:
Metrics
Y-Axis Count
Buckets
X-Axis @timestamp per hour
That would display me the results but would that be correct way?
I tried counting the dns.id field but I've noticed that i have a lot of duplicate DNS ID's for different client_ip addresses so that is definitely not the correct way.
I could also count the client_ip's but that way i will not have a timestamp, instead it will just display top 5 IPs with number of hits.
So how am i to reach the most accurate data? Is my second example good enough?
I have a DNS that receives really large traffic, so i cannot manually count it to make sure
The answer might depend on which visualization you use. But a Line visualization with the X-Axis set to Date Histogram is how I might do this.
Setting Y-Axis to Count will count all documents in the index for time period. So I think that's what you want. Or you might try setting Unique Count on the dns.id field to get rid of the duplicates.
That is how i actually have it set. I just forgot to mention that Bucket with the X-Axis is set to Date Histogram. So that's definitely good! Nice to know, thanks!!
As for the other option, if i set it to Unique count dns.id, it will leave out duplicates, yes. But there are multiple different clients (queries from client_ip) that are using the same dns.id. So those will be ignored. Don't know why. So it will not display correct data. Just for the record, if i set it to unique count dns.id it will show approx 65k hits. If i set X Axis to Date Histogram and Y to count. It will show up as 180k hits. That seems more like accurate result.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.