# Particularly large result sets **URL:** \ **Category:** Elasticsearch\ **Created:** [February 23, 2012, 7:02pm UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795 "2012-02-23T19:02:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Martin\_Bruse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_bruse/32/1276_2.png) [@Martin\_Bruse](https://discuss.elastic.co/u/Martin_Bruse)\ **Post date:** [February 23, 2012, 7:02pm UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795/1 "2012-02-23T19:02:34Z")
Hello list, I am working on a project where we will be modeling 1-n relations with potentially hundreds of thousands of endpoints for a single relationship. We are currently storing the data in Riak, and this seems to work fine. But the relations are of course a problem. Riak links are implemented in a way that limits the number of links on a single object to few thousands, so I figured I could use Riak Search to search for relationship documents of the form "user\_id: ..., follower\_id: ..." However, it seems that Riak Search scales pretty badly when you get hundreds of thousands of hits. Example: Counting the number of hits when they are 100.000 takes 10 times as longs as counting the number of hits when they are 10.000. Not too surprising, perhaps, but it makes the count take several seconds for big relations. So, I am thinking, can I do this with elasticsearch, or some other lucene based engine? This makes the question I would need an answer to this: Is counting hits for a search having hundreds of thousands of hits in elasticsearch a costly operation? Does it scale as linearly as in Riak Search? regards, //Martin ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [February 24, 2012, 1:33am UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795/2 "2012-02-24T01:33:17Z")
Hi Martin, The more docs a query matches the slower things get. The more hits you actually pull from the results (e.g. for displaying or highlighting purposes) the slower things get. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Feb 23, 2:02 pm, Martin Bruse [zondol...@gmail.com](mailto:zondol...@gmail.com) wrote: > Hello list, > > I am working on a project where we will be modeling 1-n relations with > potentially hundreds of thousands of endpoints for a single > relationship. > > We are currently storing the data in Riak, and this seems to work > fine. But the relations are of course a problem. > > Riak links are implemented in a way that limits the number of links on > a single object to few thousands, so I figured I could use Riak Search > to search for relationship documents of the form "user\_id: ..., > follower\_id: ..." > > However, it seems that Riak Search scales pretty badly when you get > hundreds of thousands of hits. Example: Counting the number of hits > when they are 100.000 takes 10 times as longs as counting the number > of hits when they are 10.000. Not too surprising, perhaps, but it > makes the count take several seconds for big relations. > > So, I am thinking, can I do this with elasticsearch, or some other > lucene based engine? > > This makes the question I would need an answer to this: Is counting > hits for a search having hundreds of thousands of hits in > elasticsearch a costly operation? Does it scale as linearly as in Riak > Search? > > regards, > //Martin ---
**Author:** ![Martin\_Bruse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_bruse/32/1276_2.png) [@Martin\_Bruse](https://discuss.elastic.co/u/Martin_Bruse)\ **Post date:** [February 24, 2012, 6:04am UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795/3 "2012-02-24T06:04:04Z")
If I don't pull all hits, just paginate through the first bit, and count them, I am basically doing what I do when I google something with millions of hits. I wonder how Google does it so fast... 😑 On Feb 24, 2012 2:33 AM, "Otis Gospodnetic" [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hi Martin, > > The more docs a query matches the slower things get. > The more hits you actually pull from the results (e.g. for displaying > or highlighting purposes) the slower things get. > > ## Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Feb 23, 2:02 pm, Martin Bruse [zondol...@gmail.com](mailto:zondol...@gmail.com) wrote: > > > Hello list, > > > > I am working on a project where we will be modeling 1-n relations with > > potentially hundreds of thousands of endpoints for a single > > relationship. > > > > We are currently storing the data in Riak, and this seems to work > > fine. But the relations are of course a problem. > > > > Riak links are implemented in a way that limits the number of links on > > a single object to few thousands, so I figured I could use Riak Search > > to search for relationship documents of the form "user\_id: ..., > > follower\_id: ..." > > > > However, it seems that Riak Search scales pretty badly when you get > > hundreds of thousands of hits. Example: Counting the number of hits > > when they are 100.000 takes 10 times as longs as counting the number > > of hits when they are 10.000. Not too surprising, perhaps, but it > > makes the count take several seconds for big relations. > > > > So, I am thinking, can I do this with elasticsearch, or some other > > lucene based engine? > > > > This makes the question I would need an answer to this: Is counting > > hits for a search having hundreds of thousands of hits in > > elasticsearch a costly operation? Does it scale as linearly as in Riak > > Search? > > > > regards, > > //Martin ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 24, 2012, 6:16am UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795/4 "2012-02-24T06:16:20Z")
Heya, ES scales very well. Counting for 10k, 100k or 1m docs will take 100 ms or less. So if you use _standard_ queries, you will get back as Google the number of hits and the 10 first relevant docs. HTH David 😉 @dadoonet Le 24 févr. 2012 à 07:04, Martin Bruse [zondolfin@gmail.com](mailto:zondolfin@gmail.com) a écrit : > If I don't pull all hits, just paginate through the first bit, and count them, I am basically doing what I do when I google something with millions of hits. I wonder how Google does it so fast... 😑 > > On Feb 24, 2012 2:33 AM, "Otis Gospodnetic" [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Hi Martin, > > The more docs a query matches the slower things get. > The more hits you actually pull from the results (e.g. for displaying > or highlighting purposes) the slower things get. > > ## Otis > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Feb 23, 2:02 pm, Martin Bruse [zondol...@gmail.com](mailto:zondol...@gmail.com) wrote: > > > Hello list, > > > > I am working on a project where we will be modeling 1-n relations with > > potentially hundreds of thousands of endpoints for a single > > relationship. > > > > We are currently storing the data in Riak, and this seems to work > > fine. But the relations are of course a problem. > > > > Riak links are implemented in a way that limits the number of links on > > a single object to few thousands, so I figured I could use Riak Search > > to search for relationship documents of the form "user\_id: ..., > > follower\_id: ..." > > > > However, it seems that Riak Search scales pretty badly when you get > > hundreds of thousands of hits. Example: Counting the number of hits > > when they are 100.000 takes 10 times as longs as counting the number > > of hits when they are 10.000. Not too surprising, perhaps, but it > > makes the count take several seconds for big relations. > > > > So, I am thinking, can I do this with elasticsearch, or some other > > lucene based engine? > > > > This makes the question I would need an answer to this: Is counting > > hits for a search having hundreds of thousands of hits in > > elasticsearch a costly operation? Does it scale as linearly as in Riak > > Search? > > > > regards, > > //Martin ---
**Author:** ![Martin\_Bruse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_bruse/32/1276_2.png) [@Martin\_Bruse](https://discuss.elastic.co/u/Martin_Bruse)\ **Post date:** [February 24, 2012, 6:18am UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795/5 "2012-02-24T06:18:14Z")
Thank you, this is what I wanted to hear 🙂 On Feb 24, 2012 7:16 AM, "David Pilato" [david@pilato.fr](mailto:david@pilato.fr) wrote: > Heya, > > ES scales very well. Counting for 10k, 100k or 1m docs will take 100 ms or > less. > > So if you use _standard_ queries, you will get back as Google the number > of hits and the 10 first relevant docs. > > HTH > David 😉 > @dadoonet > > Le 24 févr. 2012 à 07:04, Martin Bruse [zondolfin@gmail.com](mailto:zondolfin@gmail.com) a écrit : > > If I don't pull all hits, just paginate through the first bit, and count > them, I am basically doing what I do when I google something with millions > of hits. I wonder how Google does it so fast... 😑 > On Feb 24, 2012 2:33 AM, "Otis Gospodnetic" [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) > wrote: > > > Hi Martin, > > > > The more docs a query matches the slower things get. > > The more hits you actually pull from the results (e.g. for displaying > > or highlighting purposes) the slower things get. > > > > ## Otis > > > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > > > On Feb 23, 2:02 pm, Martin Bruse [zondol...@gmail.com](mailto:zondol...@gmail.com) wrote: > > > > > Hello list, > > > > > > I am working on a project where we will be modeling 1-n relations with > > > potentially hundreds of thousands of endpoints for a single > > > relationship. > > > > > > We are currently storing the data in Riak, and this seems to work > > > fine. But the relations are of course a problem. > > > > > > Riak links are implemented in a way that limits the number of links on > > > a single object to few thousands, so I figured I could use Riak Search > > > to search for relationship documents of the form "user\_id: ..., > > > follower\_id: ..." > > > > > > However, it seems that Riak Search scales pretty badly when you get > > > hundreds of thousands of hits. Example: Counting the number of hits > > > when they are 100.000 takes 10 times as longs as counting the number > > > of hits when they are 10.000. Not too surprising, perhaps, but it > > > makes the count take several seconds for big relations. > > > > > > So, I am thinking, can I do this with elasticsearch, or some other > > > lucene based engine? > > > > > > This makes the question I would need an answer to this: Is counting > > > hits for a search having hundreds of thousands of hits in > > > elasticsearch a costly operation? Does it scale as linearly as in Riak > > > Search? > > > > > > regards, > > > //Martin ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:38am UTC](https://discuss.elastic.co/t/particularly-large-result-sets/6795/6 "2017-07-06T03:38:27Z")
Closed on July 6, 2017, 3:38am UTC