# OutOfMemoryException while trying to get all distinct terms for a field **URL:** \ **Category:** Elasticsearch\ **Created:** [October 25, 2011, 3:04pm UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676 "2011-10-25T15:04:23Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Thomas\_Peuss](https://avatars.discourse-cdn.com/v4/letter/t/22d042/32.png) [@Thomas\_Peuss](https://discuss.elastic.co/u/Thomas_Peuss)\ **Post date:** [October 25, 2011, 3:04pm UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/1 "2011-10-25T15:04:23Z")
Hi! I need to get all distinct terms for a list of fields on an cluster of six nodes with 3TB of data in over 1000 indices. I use following query to get all terms for certain fields: { "query" : { "match\_all" : { } }, "facets" : { "facility" : { "terms" : { "field" : "facility", "size" : 300, "order" : "term" } }, "severity" : { "terms" : { "field" : "severity", "size" : 300, "order" : "term" } }, "hostname" : { "terms" : { "field" : "hostname", "size" : 300, "order" : "term" } }, "timestamp" : { "terms" : { "field" : "timestamp", "size" : 300, "order" : "term" } }, "mainCategory" : { "terms" : { "field" : "mainCategory", "size" : 300, "order" : "term" } }, "subCategory" : { "terms" : { "field" : "subCategory", "size" : 300, "order" : "term" } }, "country" : { "terms" : { "field" : "country", "size" : 300, "order" : "term" } } } } We get OutOfMemoryExceptions (4GB heap size) when we issue this query. How should I change the query to avoid the problem? ElasticSearch version is 0.17.8. I know that Lucene has a feature to get all distinct terms for a field without reading the whole index. How can I do that with Elasticsearch? For now we have a fixed list for the values - but that is not an "elastic" solution... 😉 CU Thomas ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2011, 3:02am UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/2 "2011-10-26T03:02:03Z")
You are using the terms facet, which causes them to be loaded to memory. There is a way to get terms in Lucene, but its not exposed in elasticsearch. On Tue, Oct 25, 2011 at 5:04 PM, Thomas Peuss [thomas.peuss@nterra.com](mailto:thomas.peuss@nterra.com)wrote: > Hi! > > I need to get all distinct terms for a list of fields on an cluster of > six nodes with 3TB of data in over 1000 indices. I use following query > to get all terms for certain fields: > > { > "query" : { > "match\_all" : { > } > }, > "facets" : { > "facility" : { > "terms" : { > "field" : "facility", > "size" : 300, > "order" : "term" > } > }, > "severity" : { > "terms" : { > "field" : "severity", > "size" : 300, > "order" : "term" > } > }, > "hostname" : { > "terms" : { > "field" : "hostname", > "size" : 300, > "order" : "term" > } > }, > "timestamp" : { > "terms" : { > "field" : "timestamp", > "size" : 300, > "order" : "term" > } > }, > "mainCategory" : { > "terms" : { > "field" : "mainCategory", > "size" : 300, > "order" : "term" > } > }, > "subCategory" : { > "terms" : { > "field" : "subCategory", > "size" : 300, > "order" : "term" > } > }, > "country" : { > "terms" : { > "field" : "country", > "size" : 300, > "order" : "term" > } > } > } > } > > We get OutOfMemoryExceptions (4GB heap size) when we issue this query. > How should I change the query to avoid the problem? Elasticsearch > version is 0.17.8. I know that Lucene has a feature to get all > distinct terms for a field without reading the whole index. How can I > do that with Elasticsearch? > > For now we have a fixed list for the values - but that is not an > "elastic" solution... 😉 > > CU > Thomas ---
**Author:** ![Thomas\_Peuss](https://avatars.discourse-cdn.com/v4/letter/t/22d042/32.png) [@Thomas\_Peuss](https://discuss.elastic.co/u/Thomas_Peuss)\ **Post date:** [October 26, 2011, 7:54am UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/3 "2011-10-26T07:54:01Z")
Hello Shay! On 26 Okt., 05:02, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > You are using the terms facet, which causes them to be loaded to memory. > There is a way to get terms in Lucene, but its not exposed in elasticsearch. So what is the way to go in elasticsearch then? CU Thomas ---
**Author:** ![phobos182](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phobos182/32/3011_2.png) [@phobos182](https://discuss.elastic.co/u/phobos182)\ **Post date:** [October 26, 2011, 1:08pm UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/4 "2011-10-26T13:08:03Z")
You can use Luke to analyze the Lucene index. [http://www.getopt.org/luke/](http://www.getopt.org/luke/) It calculates term counts and percentages. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 26, 2011, 8:38pm UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/5 "2011-10-26T20:38:06Z")
On Wed, Oct 26, 2011 at 9:54 AM, Thomas Peuss [thomas.peuss@nterra.com](mailto:thomas.peuss@nterra.com)wrote: > Hello Shay! > > On 26 Okt., 05:02, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > You are using the terms facet, which causes them to be loaded to memory. > > There is a way to get terms in Lucene, but its not exposed in > > elasticsearch. > > So what is the way to go in elasticsearch then? Building an API for terms can be a solution. It gets tricky with the distributed aspect, there used to be one, but it was removed because of the overhead in supporting it. > CU > Thomas ---
**Author:** ![Thomas\_Peuss](https://avatars.discourse-cdn.com/v4/letter/t/22d042/32.png) [@Thomas\_Peuss](https://discuss.elastic.co/u/Thomas_Peuss)\ **Post date:** [October 27, 2011, 7:20am UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/6 "2011-10-27T07:20:23Z")
Hi! On 26 Okt., 15:08, phobos182 [phobos...@gmail.com](mailto:phobos...@gmail.com) wrote: > You can use Luke to analyze the Lucene index. > > [http://www.getopt.org/luke/](http://www.getopt.org/luke/) > > It calculates term counts and percentages. I know Luke. This is not an option. We need to get the terms without generating a terms list by hand out of one thousand indices... CU Thomas ---
**Author:** ![Thomas\_Peuss](https://avatars.discourse-cdn.com/v4/letter/t/22d042/32.png) [@Thomas\_Peuss](https://discuss.elastic.co/u/Thomas_Peuss)\ **Post date:** [October 27, 2011, 7:25am UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/7 "2011-10-27T07:25:17Z")
Hi Shay! On 26 Okt., 22:38, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > So what is the way to go in elasticsearch then? > > Building an API for terms can be a solution. It gets tricky with the > distributed aspect, there used to be one, but it was removed because of the > overhead in supporting it. IMHO it is OK if such an API would deliver not a distinct list if that is the problem. We could handle that in our application and the list of distinct terms is very limited in our case (which is part of the problem because many rows share the same terms). We don't want to compile such a list by hand. It would be outdated when we are finished compiling it. 😉 CU Thomas ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:50am UTC](https://discuss.elastic.co/t/outofmemoryexception-while-trying-to-get-all-distinct-terms-for-a-field/5676/8 "2017-07-06T03:50:43Z")
Closed on July 6, 2017, 3:50am UTC