# How can I get accurate unique count from elasticsearch?

**URL:** <https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397>\
**Category:** Elasticsearch\
**Created:** [April 24, 2017, 10:11am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397 "2017-04-24T10:11:19Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lyncean\_Cura](https://avatars.discourse-cdn.com/v4/letter/l/ed655f/32.png) [@Lyncean\_Cura](https://discuss.elastic.co/u/Lyncean_Cura)\
**Post date:** [April 24, 2017, 10:11am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/1 "2017-04-24T10:11:19Z")

</div>

I also used {"precision\_threshold": 1000} but still don't get the correct count.

---

<div class="post-metadata">

**Author:** ![ultimate.duwal](https://avatars.discourse-cdn.com/v4/letter/u/2acd7d/32.png) [@ultimate.duwal](https://discuss.elastic.co/u/ultimate.duwal)\
**Post date:** [April 24, 2017, 10:20am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/2 "2017-04-24T10:20:47Z")

</div>

The max threshold supported by elasticsearch is 40000. Try it. But it sometimes doesn't give the required result as expected or say accurate with other more expensive queries.

---

<div class="post-metadata">

**Author:** ![Lyncean\_Cura](https://avatars.discourse-cdn.com/v4/letter/l/ed655f/32.png) [@Lyncean\_Cura](https://discuss.elastic.co/u/Lyncean_Cura)\
**Post date:** [April 24, 2017, 11:01am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/3 "2017-04-24T11:01:16Z")

</div>

I tried, but not able to get the correct result.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [April 24, 2017, 11:07am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/4 "2017-04-24T11:07:28Z")

</div>

What is the expected count and the returned count in this case?

---

<div class="post-metadata">

**Author:** ![Lyncean\_Cura](https://avatars.discourse-cdn.com/v4/letter/l/ed655f/32.png) [@Lyncean\_Cura](https://discuss.elastic.co/u/Lyncean_Cura)\
**Post date:** [April 24, 2017, 11:24am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/5 "2017-04-24T11:24:17Z")

</div>

Expected: 3921  
Actual: 3671

I have visit documents which have visiti\_id and visitor\_id, I tried to find out the No of visitors visited.  
So result should be like for 5000 (count of visit\_id) visits, number of visitors are 3921(unique count of visitor\_id)  
I am using KIBI for data display.

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [April 24, 2017, 11:27am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/6 "2017-04-24T11:27:18Z")

</div>

OK. And what were the numbers were when you tried the max precision threshold of 40,000?

---

<div class="post-metadata">

**Author:** ![Lyncean\_Cura](https://avatars.discourse-cdn.com/v4/letter/l/ed655f/32.png) [@Lyncean\_Cura](https://discuss.elastic.co/u/Lyncean_Cura)\
**Post date:** [April 24, 2017, 11:56am UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/7 "2017-04-24T11:56:11Z")

</div>

without precision\_threshold: = 3689  
{"precision\_threshold": 1000} = 3698  
{"precision\_threshold": 40000} = 3703

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [April 24, 2017, 2:09pm UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/8 "2017-04-24T14:09:05Z")

</div>

This smells fishy. There's a small possibility of error when set to safe mode (actual value counts are \<= precision\_threshold) and this comes down to the implementation which is based on counting unique hash codes. Hash collisions are rare but possible which means a possibly small amount of undercounting but the scale of your expected vs returned discrepancies (3921 vs 3703) seems off.

Can you share more about your query and mappings? How many buckets do you get if you use a `terms` agg on the visitor\_id field with `size` set to 40,000? Are you sure the expected number of 3,921 is correct. I appreciate some of the data may be too large or sensitive to share publicly so feel free to message me directly with zipped results if that helps.

Cheers  
Mark

---

<div class="post-metadata">

**Author:** ![Lyncean\_Cura](https://avatars.discourse-cdn.com/v4/letter/l/ed655f/32.png) [@Lyncean\_Cura](https://discuss.elastic.co/u/Lyncean_Cura)\
**Post date:** [April 24, 2017, 3:42pm UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/9 "2017-04-24T15:42:32Z")

</div>

Thanks @Mark_Harwood for the reply.  
I will recheck the data integrity.  
Also I can't share the actual data with you so I will create the similar dummy data and try to reproduce the same issue. If the issue reproduces then I will share the test data with you.

Also I forget to mention that I filter the result first with some flag value.

Elasticsearch request body:  
{  
"query": {  
"filtered": {  
"query": {  
"query\_string": {  
"query": "booked\_by\_app:Yes",  
"analyze\_wildcard": true  
}  
},  
"filter": {  
"bool": {  
"must": [  
{  
"range": {  
"booking\_date": {  
"gte": 1477324276000,  
"lte": 1493049076000,  
"format": "epoch\_millis"  
}  
}  
}  
],  
"must\_not": []  
}  
}  
}  
},  
"size": 0,  
"aggs": {  
"2": {  
"cardinality": {  
"field": "visitor",  
"precision\_threshold": 1000  
}  
}  
}  
}Preformatted text

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2017, 3:56pm UTC](https://discuss.elastic.co/t/how-can-i-get-accurate-unique-count-from-elasticsearch/83397/10 "2017-05-22T15:56:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.