Describe the issue: Inconsistency count observed for term aggregation operation
We are applying aggregation on keyword type field (non-nested)
our mapping
{
"mappings": {
"dynamic": "strict",
"properties": {
"os": {
"properties": {
"name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"copy_to": [
"fullText"
]
}
}
}
}
}
}
our query
{
"size": 0,
"aggregations": {
"os.name.raw-TERMS": {
"terms": {
"field": "os.name.raw",
"size": 10,
// "shard_size" -> maybe 15 to 20
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"_count": "desc"
},
{
"_key": "asc"
}
]
}
}
}
}
response
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 15,
"successful": 15,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 10000,
"relation": "gte"
},
"max_score": null,
"hits": []
},
"aggregations": {
"os.name.raw-TERMS": {
"doc_count_error_upper_bound": 72,
"sum_other_doc_count": 2631,
"buckets": [
{
"key": "Unidentified",
"doc_count": 25349
},
... 6 document present
{
"key": "Microsoft Windows",
"doc_count": 74
},
... 2 document present
]
}
}
}
as you can see in aggregation bucket
“Microsoft Windows” doc_count is 74
if we increase shard size >= 1000, we get accurate result
because doc_count_error_upper_bound is 0
i checked the internal working from documentation that shard_size is number of buckets created per shards before merging into coordinate node to provide result.
and doc_count_error_upper_bound :0 means results are now accurate.
ISSUE!
here we have a issue.
i checked the uniqueness count with cardinality aggregation for os.name.raw it is 478, so anything above this in shard_size will make doc_count_error_upper_bound as 0.
but it is not limited for a single index but for multiple index we need TOP N result, we cant check its uniqueness for all before querying it.
If we put a higher cap on shard_size, it is still possible that uniqueness may increase in the future and cross that value too. So whatever value we put, there is no guarantee that it will always produce accurate results.
We are sure that data uniqueness will continue to increase in the index over time.
This means we will have to keep adjusting shard_size repeatedly to maintain accuracy.
We want to avoid manually adjusting this value or using very high values because it causes heavy computation and performance overhead.
NEED
I want to know if any other aggregation or way to get top N document document with accurate count.
we want to sort document in desc order and keeping keys in ascending.
In this order we need result
{
“_count”: “desc”
},
{
“_key”: “asc”
}
#Already check composite agg, sort by _count is not supported
Please help me how to resolve this issue or if there is a recommended approach for this scenario.
Thanks!