# SignificantTerms Agg : \_superset\_size greater than doc\_count !? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 17, 2015, 2:54pm UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486 "2015-09-17T14:54:38Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![tomlameche](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomlameche/32/4832_2.png) [@tomlameche](https://discuss.elastic.co/u/tomlameche)\ **Post date:** [September 17, 2015, 2:54pm UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/1 "2015-09-17T14:54:38Z")
Hello, My appology for my bad english... I'm trying to script a custom score for significant aggregation, and i'm very surprised with the value off the variables \_superset\_freq and \_superset\_size. =\> \_superset\_size is greater than doc\_count : very strange no ? =\> \_superset\_freq for each buckets is greater than a simple count with term aggregation for the same term I don't understand what happened... Does anybody have an explanation ? Perhaps there is something i do wrong... Exemple : Thanks, Tom ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [September 17, 2015, 4:15pm UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/2 "2015-09-17T16:15:25Z")
"subset" relates to the docs that match your query/parent bucket in the agg tree. "superset" relates to the index from which these are drawn (or your choice of background\_filter). The stats in significant\_terms calculations essentially perform a diff between popularity of terms in the subset and their popularity in the superset. Does this help clarify? ---
**Author:** ![tomlameche](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomlameche/32/4832_2.png) [@tomlameche](https://discuss.elastic.co/u/tomlameche)\ **Post date:** [September 17, 2015, 5:28pm UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/3 "2015-09-17T17:28:05Z")
yes, the definition of subset and superset is clear. My problem is that the value of \_superset\_size in a script is greater than the total number of document in my index. I think there is a bug somewhere. I do a nested significant terms aggregation after a simple term aggregation, similar as the exemple describe here : [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html) I need to perform a custom score and so i try to used "script\_heuristic" : the result seems strange for me, so i modify my script to view the value of each variable (\_superset\_size, \_superset\_freq, \_subset\_freq and \_subsetset\_size) with : "script\_heuristic": { "script": "\_superset\_size" } And what a surprise : the value of \_superset\_size is greater than the number of total document in my index... In addition, the value of bg\_count is greater than the value of total count for each terms, as descibe here [Bg\_counts in nested significant\_terms aggregation](https://discuss.elastic.co/t/bg-counts-in-nested-significant-terms-aggregation/21878/2) There is a bug, i guess ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [September 17, 2015, 5:38pm UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/4 "2015-09-17T17:38:51Z")
Are you using any of the following: - nested docs? - aliases with filters? - indexes with deleted docs? - multiple indices/shards where some of them do not contain fields or content related to the query? They all have the potential to skew the stats. The fact that these stats are approximate is acknowledged here: [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html#\_approximate\_counts](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html#_approximate_counts) ---
**Author:** ![tomlameche](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomlameche/32/4832_2.png) [@tomlameche](https://discuss.elastic.co/u/tomlameche)\ **Post date:** [September 18, 2015, 7:42am UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/5 "2015-09-18T07:42:57Z")
Thanks for your help, yes i'm using nested docs, and perhaps indexes with deleted docs. So, it's not a bug, it's a feature 😉 ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [September 18, 2015, 8:28am UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/6 "2015-09-18T08:28:16Z")
We use the fastest source of stats which is the pre-computed counts held in the Lucene index which are susceptible to the accuracy issues I outlined. I'd probably add to the list of gotchas the situation when you have multiple document types in the same index. However, it is possible to define an alternative source of background stats which relies on re-counting values on the fly for all docs that match a given filter [1]. This may provide a way to fix the accuracy issues you are experiencing. [1] [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html#\_custom\_background\_context](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-significantterms-aggregation.html#_custom_background_context) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:49pm UTC](https://discuss.elastic.co/t/significantterms-agg--superset-size-greater-than-doc-count/29486/7 "2017-07-05T23:49:29Z")