# Counting distinct terms in field **URL:** \ **Category:** Elasticsearch\ **Created:** [July 30, 2011, 5:21pm UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003 "2011-07-30T17:21:57Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Daniel\_E](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_e/32/3137_2.png) [@Daniel\_E](https://discuss.elastic.co/u/Daniel_E)\ **Post date:** [July 30, 2011, 5:21pm UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003/1 "2011-07-30T17:21:57Z")
We have a field called signature.full which is unanalyzed, every document has a single string which represents one possible term. We are displaying a page that shows a table of unique signatures, ordered by frequency. Unfortunately, there is no good way that we have found to see how many unique signatures there are other than doing a facet for an absurdly high range. "facets" : { "signatures" : { "terms" : { "field" : "signature.full", "size" : MAXINT } } } MAXINT being 2^32 because ES runs in a 32 bits machine. And MAXINT is just a trick to get all the results so we can count it. Is there a better way to do this? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 30, 2011, 6:01pm UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003/2 "2011-07-30T18:01:27Z")
No, there isn't another way to do it. Sadly, distinct is something that is hard to do in a distributed env for large result sets. We can easily count the distinct count per shard, but to return a correct number across shards menas we need to send all the (distinct) values back to the the "coordinator" and compute the distinct count there... . On Sat, Jul 30, 2011 at 8:21 PM, Daniel E [deinspanjer@gmail.com](mailto:deinspanjer@gmail.com) wrote: > We have a field called signature.full which is unanalyzed, every > document has a single string which represents one possible term. > > We are displaying a page that shows a table of unique signatures, > ordered by frequency. Unfortunately, there is no good way that we > have found to see how many unique signatures there are other than > doing a facet for an absurdly high range. > > "facets" : { > "signatures" : { > "terms" : { > "field" : "signature.full", > "size" : MAXINT > } > } > } > > MAXINT being 2^32 because ES runs in a 32 bits machine. And MAXINT is > just a trick to get all the results so we can count it. > > Is there a better way to do this? ---
**Author:** ![Liyu](https://avatars.discourse-cdn.com/v4/letter/l/ba9def/32.png) [@Liyu](https://discuss.elastic.co/u/Liyu)\ **Post date:** [September 20, 2011, 1:17am UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003/3 "2011-09-20T01:17:57Z")
If we want to implement a reduce function on the large result set, will you have any suggestions for a clean and elegant solution (something similar to what MongoDB does), or does ES have this feature on the roadmap? Thanks a bunch. On Jul 30, 11:01 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > No, there isn't another way to do it. Sadly, distinct is something that is > hard to do in a distributed env for large result sets. We can easily count > the distinct count per shard, but to return a correct number across shards > menas we need to send all the (distinct) values back to the the > "coordinator" and compute the distinct count there... . > > On Sat, Jul 30, 2011 at 8:21 PM, Daniel E [deinspan...@gmail.com](mailto:deinspan...@gmail.com) wrote: > > > We have a field called signature.full which is unanalyzed, every > > document has a single string which represents one possible term. > > > We are displaying a page that shows a table of unique signatures, > > ordered by frequency. Unfortunately, there is no good way that we > > have found to see how many unique signatures there are other than > > doing a facet for an absurdly high range. > > > "facets" : { > > "signatures" : { > > "terms" : { > > "field" : "signature.full", > > "size" : MAXINT > > } > > } > > } > > > MAXINT being 2^32 because ES runs in a 32 bits machine. And MAXINT is > > just a trick to get all the results so we can count it. > > > Is there a better way to do this? ---
**Author:** ![Liyu](https://avatars.discourse-cdn.com/v4/letter/l/ba9def/32.png) [@Liyu](https://discuss.elastic.co/u/Liyu)\ **Post date:** [September 20, 2011, 5:25am UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003/4 "2011-09-20T05:25:33Z")
BTW, I am from Symantec and we are in the process evaluating ES. Thanks, -- Liyu On Mon, Sep 19, 2011 at 6:17 PM, Liyu [liyuyi@gmail.com](mailto:liyuyi@gmail.com) wrote: > If we want to implement a reduce function on the large result set, > will you have any suggestions for a clean and elegant solution > (something similar to what MongoDB does), or does ES have this feature > on the roadmap? > > Thanks a bunch. > > On Jul 30, 11:01 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > No, there isn't another way to do it. Sadly, distinct is something that > > is > > hard to do in a distributed env for large result sets. We can easily > > count > > the distinct count per shard, but to return a correct number across > > shards > > menas we need to send all the (distinct) values back to the the > > "coordinator" and compute the distinct count there... . > > > > On Sat, Jul 30, 2011 at 8:21 PM, Daniel E [deinspan...@gmail.com](mailto:deinspan...@gmail.com) wrote: > > > > > We have a field called signature.full which is unanalyzed, every > > > document has a single string which represents one possible term. > > > > > We are displaying a page that shows a table of unique signatures, > > > ordered by frequency. Unfortunately, there is no good way that we > > > have found to see how many unique signatures there are other than > > > doing a facet for an absurdly high range. > > > > > "facets" : { > > > "signatures" : { > > > "terms" : { > > > "field" : "signature.full", > > > "size" : MAXINT > > > } > > > } > > > } > > > > > MAXINT being 2^32 because ES runs in a 32 bits machine. And MAXINT is > > > just a trick to get all the results so we can count it. > > > > > Is there a better way to do this? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 20, 2011, 8:36am UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003/5 "2011-09-20T08:36:26Z")
Yes, something like that is planned to be implemented. Not sure if there is an issue for it or not. On Tue, Sep 20, 2011 at 4:17 AM, Liyu [liyuyi@gmail.com](mailto:liyuyi@gmail.com) wrote: > If we want to implement a reduce function on the large result set, > will you have any suggestions for a clean and elegant solution > (something similar to what MongoDB does), or does ES have this feature > on the roadmap? > > Thanks a bunch. > > On Jul 30, 11:01 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > No, there isn't another way to do it. Sadly, distinct is something that > > is > > hard to do in a distributed env for large result sets. We can easily > > count > > the distinct count per shard, but to return a correct number across > > shards > > menas we need to send all the (distinct) values back to the the > > "coordinator" and compute the distinct count there... . > > > > On Sat, Jul 30, 2011 at 8:21 PM, Daniel E [deinspan...@gmail.com](mailto:deinspan...@gmail.com) wrote: > > > > > We have a field called signature.full which is unanalyzed, every > > > document has a single string which represents one possible term. > > > > > We are displaying a page that shows a table of unique signatures, > > > ordered by frequency. Unfortunately, there is no good way that we > > > have found to see how many unique signatures there are other than > > > doing a facet for an absurdly high range. > > > > > "facets" : { > > > "signatures" : { > > > "terms" : { > > > "field" : "signature.full", > > > "size" : MAXINT > > > } > > > } > > > } > > > > > MAXINT being 2^32 because ES runs in a 32 bits machine. And MAXINT is > > > just a trick to get all the results so we can count it. > > > > > Is there a better way to do this? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:54am UTC](https://discuss.elastic.co/t/counting-distinct-terms-in-field/5003/6 "2017-07-06T03:54:05Z")