# How to sort on small cluster with 100m+ documents without OOME **URL:** \ **Category:** Elasticsearch\ **Created:** [April 15, 2013, 10:41am UTC](https://discuss.elastic.co/t/how-to-sort-on-small-cluster-with-100m-documents-without-oome/11584 "2013-04-15T10:41:48Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Marek\_Skorek](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Marek\_Skorek](https://discuss.elastic.co/u/Marek_Skorek)\ **Post date:** [April 15, 2013, 10:41am UTC](https://discuss.elastic.co/t/how-to-sort-on-small-cluster-with-100m-documents-without-oome/11584/1 "2013-04-15T10:41:48Z")
Hi, I have to query my data on sort them by some field (datetime for example). If I query the indexes with various criteria there is no problem until a sorting is added. I was expected to have the filtered data sorted (matched ~1k of 100m+) but it seems the engine tries to load all the field values into the memory which causes OOME. The question is if there is an opportunity of sorting only the filtered data only? (I do not store the document's source) Do you have an experience in this case? Really thanks for any advices. Regards. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mark\_Harwood1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood1/32/101255_2.png) [@Mark\_Harwood1](https://discuss.elastic.co/u/Mark_Harwood1)\ **Post date:** [April 16, 2013, 2:32pm UTC](https://discuss.elastic.co/t/how-to-sort-on-small-cluster-with-100m-documents-without-oome/11584/2 "2013-04-16T14:32:09Z")
I'm somewhat pessimistic about sorting, from both a technical and a user-experience standpoint: - Technically - if you're not caching masses of values in RAM (your current problem) you're having to hit disk randomly to retrieve these values for a large number of matching docs (which is slow). - From a user-experience point of view: a lot of what Lucene has to offer is in relevance-ranked ordering of partial matches i.e. it is designed to produce fuzzy sets of results where, unlike databases, membership of the set is measured to a degree. Sorting a fuzzy set, (or for that matter, producing facet counts) only helps surface the low-quality matches that may otherwise be lurking unseen at the long-tail end of that very large fuzzy set. Sometimes, you can solve the user's problem a different way e.g. if you are sorting most-recent-first because you want to let a user see current content then consider the Google search approach instead - they offer users a range of filters ("past hour", "past week" etc). These are technically more efficient to implement and users benefit from the natural relevancy ranking order keeping highest-quality matches first and the long-tail of low-quality crud out of sight. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:40am UTC](https://discuss.elastic.co/t/how-to-sort-on-small-cluster-with-100m-documents-without-oome/11584/3 "2017-07-06T02:40:56Z")
Closed on July 6, 2017, 2:40am UTC