# Python3 Large Query Cardinality & Query return different results

**URL:** <https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717>\
**Category:** Elasticsearch\
**Created:** [November 17, 2020, 2:36pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717 "2020-11-17T14:36:37Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![stcdarrell](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\
**Post date:** [November 17, 2020, 2:36pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717/1 "2020-11-17T14:36:37Z")

</div>

> hi, i'm using python3 and the elasticsearch library to query some very large ES indexes.
> 
> I've been following this blog post as a reference:
> 
> > **[Learn how to use scroll Elasticsearch aggregation](https://lukasmestan.com/learn-how-to-use-scroll-elasticsearch-aggregation/)**
> >
> > Learn how to use scroll Elasticsearch aggregation
> 
> I'm getting some results i dont understand, the Cardnality returns a certain number ("precision\_threshold": 100) and when i page through the aggrigration i get a different number of results returned:
> 
> Cardinality: 41941  
> Results Returned: 41084  
> .. so there is a difference in 857.. those results are important
> 
> Any suggestions would be appreciated.
> 
> Basic Code i'm using below:
> 
> for getting the cardinality:  
> ``
> 
> `
> ```
> #creates the es object
> es = Elasticsearch(hosts=[self.host], timeout=60, max_retries=3, retry_on_timeout=True)
> dataDict = {}
> 
> #this gets a rough estimate of how many records will be returned
> page = es.search(
> index=self.index,
> scroll='20m',
> body={
> "aggs": {
> "type_count": {
> "cardinality": {
> "field": self.field,
> "precision_threshold": 100
> } #end cardinality
> }#end type count
> }#end aggs
> }#end body
> )
> #print(page)
> print("Cardinality Page Results:", page['aggregations']['type_count']['value'])
> unique_results = page['aggregations']['type_count']['value']
> page_size = 10000
> 
> pages_needed = unique_results / page_size
> pages_used=math.ceil(pages_needed)
> print ("Math:", pages_needed, " : ", "Rounded", pages_used )
> agg_no_pages=pages_used
> 
> ```
> `
> 
> This is the code i'm using to retrieve the results:
> 
> `
> ```
> page = es.search(
> index=self.index,
> scroll='20m',
> size=10000,
> body={
> "size": 0,
> "aggs": {
> "unique_ip": {
> "terms": {
> "field": self.field,
> "include":{
> "partition":0,
> "num_partitions": agg_no_pages
> }, #end include
> "size": page_size
> },
> } #end unique IP
> } #end aggs
> })
> 
> #print (page['aggregations']['unique_ip'])
> print (page['aggregations']['unique_ip']['sum_other_doc_count'])
> print ("--===================--")
> for i in range(agg_no_pages):
> print ("Round:", i)
> page = es.search(
> index=self.index,
> scroll='2m',
> size=10000,
> body={
> "size": 0,
> "aggs": {
> "unique_ip": {
> "terms": {
> "field": self.field,
> "include": {
> "partition": i,
> "num_partitions": agg_no_pages
> }, # end include
> "size": 10000
> },
> "aggs": {
> "ip_count": {
> "cardinality": {
> "field": self.field
> }
> }
> }
> }
> }
> })
> #print(page['aggregations']['unique_ip'])
> #print(page['aggregations']['unique_ip']['sum_other_doc_count'])
> for item in page['aggregations']['unique_ip']['buckets']:
> #print ("Item:", item['key'], " : ", item['doc_count'])
> dataDict[item['key']]=item['doc_count']
> 
> return dataDict.copy()
> 
> ```
> `

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [November 17, 2020, 2:44pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717/2 "2020-11-17T14:44:12Z")

</div>

This is the logic of [cardinality aggs](https://elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html), it will never give you 100% of all occurences of your term. you can increase precision\_threshold up to 40000, but you will never get 100% 🙂

---

<div class="post-metadata">

**Author:** ![stcdarrell](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\
**Post date:** [November 17, 2020, 3:12pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717/3 "2020-11-17T15:12:35Z")

</div>

thank you, thats what i was hoping.. i just needed someone else to let me know that was alright.

Is this the right approach to retrieving data?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [November 17, 2020, 3:23pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717/4 "2020-11-17T15:23:16Z")

</div>

I used to deal with same use case with high cardinality where i need to scroll all occurences  
The best approche for me was [pagination](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_filtering_values_with_partitions)

---

<div class="post-metadata">

**Author:** ![stcdarrell](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@stcdarrell](https://discuss.elastic.co/u/stcdarrell)\
**Post date:** [November 17, 2020, 6:45pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717/5 "2020-11-17T18:45:44Z")

</div>

thank you, thats what i'm doing

thank you for your help

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 15, 2020, 6:45pm UTC](https://discuss.elastic.co/t/python3-large-query-cardinality-query-return-different-results/255717/6 "2020-12-15T18:45:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.