# Cardinality Aggregation gives wrong number? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 5, 2019, 12:15pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116 "2019-02-05T12:15:55Z")\ **Posts on this page:** 14\ **Page:** 2
**Author:** ![Wassim\_Ben\_Amor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wassim_ben_amor/32/39799_2.png) [@Wassim\_Ben\_Amor](https://discuss.elastic.co/u/Wassim_Ben_Amor)\ **Post date:** [February 6, 2019, 2:39pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/21 "2019-02-06T14:39:21Z")
@Mark_Harwood I'm still blocked .. Can you tell me what is exactly the precision\_threshold ? Because I can't find the technical part of it in the docs. Is it when it's set to 40k for example, means when calculating cardinality of a set containing \<40k, it will be exact ?? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 6, 2019, 2:52pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/22 "2019-02-06T14:52:09Z")
Yes, you can think of it as the threshold between when it switches between stuffing values into a hashmap (which is an accurate way to count) versus using a probabilistic data structure which is inaccurate but more scalable ---
**Author:** ![Wassim\_Ben\_Amor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wassim_ben_amor/32/39799_2.png) [@Wassim\_Ben\_Amor](https://discuss.elastic.co/u/Wassim_Ben_Amor)\ **Post date:** [February 6, 2019, 3:04pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/23 "2019-02-06T15:04:55Z")
So, to resume all this. There is no way to get exact cardinality value. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 6, 2019, 3:13pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/24 "2019-02-06T15:13:44Z")
Ah no - I think I see what you've done. You've got hash collisions. The hash of the `userID` shouldn't be held in a field called `userID` - it should be in a field called `userIDHash` (for that is what it is). You use `userIDHash` to group results into smaller work partitions for processing (using that script query) but you always do your cardinality agg on the `userID` field. ---
**Author:** ![Wassim\_Ben\_Amor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wassim_ben_amor/32/39799_2.png) [@Wassim\_Ben\_Amor](https://discuss.elastic.co/u/Wassim_Ben_Amor)\ **Post date:** [February 6, 2019, 4:37pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/25 "2019-02-06T16:37:27Z")
Well, I assume that my Hashing algorithm (32bits hash from farmhash(node library)) has made some collisions. So for testing I have deleted user Ids which makes collisions (6 user ids). So I have to find exactly 192 625 unique user ID. I have made the field user\_id\_hash even if i'm not convinced, because calculating cardinality on hash or the string value is the same, because they must be uniq. So like always, I have run my script ``` var N = 20; var arr = Array.apply(null, { length: N }).map(Number.call, Number) console.log(arr) client.msearch({ body: [].concat.apply([], arr.map((number) => ([ { index: 'my-index' }, { "size": 0, "query": { "bool": { "filter": { "script": { "script": { "source": "doc['user_id_hash'].value %" + N + " == params.param1", "lang": "painless", "params": { "param1": number } } } } } }, "aggs": { "numberOfUsers": { "cardinality": { "field": "final_user_id", "precision_threshold": 40000 } } } } ]))) }, (err, result) => { if (err) { console.log(err) } else { console.log("TOTAL CARDINALITY") console.log(result.responses.map(e => e.aggregations.numberOfUsers.value).sum()) console.log("CARDINALITIES PER PARTITION") console.log(result.responses.map(e => e.aggregations.numberOfUsers.value)) console.log("DOC COUNT PER PARTITION") console.log(result.responses.map(e => e.hits.total)) } }) ``` which gives the wrong cardinality: ``` TOTAL CARDINALITY 192618 CARDINALITIES PER PARTITION [ 9569, 9568, 9563, 9711, 9436, 9485, 9560, 9808, 9703, 9517, 9732, 9678, 9594, 9718, 9721, 9602, 9670, 9542, 9816, 9625 ] DOC COUNT PER PARTITION [ 25507, 25941, 26224, 24981, 25582, 25108, 26235, 25508, 26982, 26750, 26999, 26178, 25190, 25702, 26841, 25250, 26357, 26333, 27842, 28144 ] ``` I'm really not convinced by the precision\_threshold, it's like random calculation. Event if I have tried to simulate the repartition and calculate the cardinalities manually, I can see that the cardinality calculated in each repartition is different of the result of ES calculation. code : ``` let usersOrignal = JSON.parse(fs.readFileSync('./users-original.json')) let usersHash = usersOrignal.map(e => hash.hash32(e)) console.log("CORRECT CARDINALITY :") console.log([...new Set(usersHash)].length) console.log(usersHash.length) var N = 20; var arr = Array.apply(null, { length: N }).map(Number.call, Number) console.log("CORRECT CARDINALITIES :") console.log(arr.map(number => { return (usersHash.filter(e => e % N == number).length) })) ``` which gives this output : ``` CORRECT CARDINALITY : 192625 192625 CORRECT CARDINALITIES : [ 9570, 9568, 9563, 9711, 9435, 9484, 9560, 9808, 9703, 9517, 9735, 9678, 9594, 9719, 9723, 9603, 9671, 9542, 9815, 9626 ] ``` I really can't understand ... ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 6, 2019, 5:37pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/26 "2019-02-06T17:37:04Z")
How many shards are you using? Does it fail with only 1 shard? ---
**Author:** ![Wassim\_Ben\_Amor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wassim_ben_amor/32/39799_2.png) [@Wassim\_Ben\_Amor](https://discuss.elastic.co/u/Wassim_Ben_Amor)\ **Post date:** [February 6, 2019, 5:42pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/27 "2019-02-06T17:42:25Z")
I am using one shard, ``` "number_of_shards" : "1", "number_of_replicas" : "0", ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 6, 2019, 5:47pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/28 "2019-02-06T17:47:43Z")
OK - if you can identify an example of \<40k IDs that are counted incorrectly then it sounds like we can open a bug for that. What version of elasticsearch? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 6, 2019, 6:05pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/29 "2019-02-06T18:05:14Z")
I mis-spoke. The precision threshold _is_ a distinction between using method 1 and method 2 for counting but method 1 still does not guarantee absolute accuracy - it can still be subject to hash collisions. Apologies for the misleading advice. ---
**Author:** ![Wassim\_Ben\_Amor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wassim_ben_amor/32/39799_2.png) [@Wassim\_Ben\_Amor](https://discuss.elastic.co/u/Wassim_Ben_Amor)\ **Post date:** [February 7, 2019, 9:42am UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/30 "2019-02-07T09:42:16Z")
So, how can we define the threshold exactly ? it is supposed to give exact value (set to 40k) when documents are under 40k ? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 7, 2019, 10:08am UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/31 "2019-02-07T10:08:13Z")
> [@Wassim\_Ben\_Amor](#): > > It is supposed to give exact value (set to 40k) when documents are under 40k ? No, I assumed wrong. Under the precision threshold (which is max 40k) it uses one counting technique and if the buckets collected gets greater than this number it switches to a cheaper but less accurate technique. Even the first technique has a small margin of error. For fully accurate counting of large numbers of IDs you can use either the [composite aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html) with the `after` parameter or the slightly messier "terms aggregation with partitioning" approach you used [here](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/10). In each case your client will have to page through to completion and count the matches. I appreciate the confusion in the array of choices available. It's impossible to satisfy all requirements around speed, accuracy and scale so there are a number of techniques possible each with different trade-offs. I've been sketching out ideas for a blog on the topic to help people navigate the options which is centred around a decision tree something like this: ![So%20you%20want%20to%20GROUP%20BY](https://us1.discourse-cdn.com/elastic/original/3X/f/d/fd5cdcc2aaa3a4e02e8dd9ae8de19231328a7597.png) I'd appreciate any feedback on if this sort of thing would be useful. ---
**Author:** ![Wassim\_Ben\_Amor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wassim_ben_amor/32/39799_2.png) [@Wassim\_Ben\_Amor](https://discuss.elastic.co/u/Wassim_Ben_Amor)\ **Post date:** [February 7, 2019, 11:32am UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/32 "2019-02-07T11:32:57Z")
Thank you for your reply. Just a last notice. I have tried to see the cardinality on a little dataset (18k of documents containing userId field), and it makes also an approximation. For real, i am disappointed that elastic can not produce an exact analytics result even on low data size. At least, now, i'm sure Elastic is not made for analytics, I need a better big data solution. Thank you for your time. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [February 7, 2019, 12:10pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/33 "2019-02-07T12:10:36Z")
> [@Wassim\_Ben\_Amor](#): > > i am disappointed that elastic can not produce an exact analytics result even on low data size. Everything elasticsearch does is designed for scale. We don't want to build data analysis functions that blow up when users provide us with a lot of data. > [@Wassim\_Ben\_Amor](#): > > i'm sure Elastic is not made for analytics, I need a better big data solution. The world of big data is, by necessity, built on fuzzier constructs [1]. We're using the same algorithms used by the other big data platforms for exactly the same reasons. As I outlined [here](https://discuss.elastic.co/t/background-count-in-significant-terms-not-consistent/55824/8) - it's a necessary trade off in the age of big data. You can't expect to beat physics. Arguably we could offer a function to guarantee cardinality accuracy at small scale but small scale is not our mission. [1] [Introduction to Probabilistic Data Structures - DZone](https://dzone.com/articles/introduction-probabilistic-0) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2019, 12:10pm UTC](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116/34 "2019-03-07T12:10:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/cardinality-aggregation-gives-wrong-number/167116.md?page=1)