Hello All,
I am new to Elasticsearch and this is my first post so please be kind. I am using Elastic version 7.6 and I'm collecting Suricata IDS data with the filebeat plugin. I'm interested in a field called dns.question.name which I plan on pulling from elasticsearch and enriching the domain data with reputation data from a 3rd party API.
I have no problems with the data but I am having problems figuring out my query approach.
In this query, I get the exact data I am looking for which is unique domains in aggregate:
GET filebeat-7.6.1*/_search
{
"query" : {
"match_all" : { }
},
"aggregations" : {
"label_agg" : {
"terms" : {
"field" : "dns.question.name",
"size" : 10
}
}
}
}
The problem is that I am getting hundreds of records per second and I run into issues with exceeding the max bucket size when run a query with a size over 5,000. What I was hoping to do was to write a range based query that pulled the aggregates from a smaller time window so I don't hit the max bucket thresholds.
Has anyone had any experience doing this? If so how did you solve it?
Or, can anyone point me in the right direction on how to implement range based searching on this aggregate query? I have spent tons of time in the documentation but I just can find to seem the right syntax for it.
Note, I wrote this range based query but I can't seem to get the write syntax to pull only the dns.question.name field from it:
GET filebeat-7.6.1*/_search?
{
"query":{
"range":{
"suricata.eve.timestamp":{
"gte":"now-10s",
"lt":"now"
}
}
},
"_source" :
{
}
}
Any help you can give me would be most appreciated!
-Dustin