# Always view all results from search? (Java Client) **URL:** \ **Category:** Elasticsearch\ **Created:** [April 2, 2012, 2:11pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202 "2012-04-02T14:11:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Drew\_H\_2](https://avatars.discourse-cdn.com/v4/letter/d/77aa72/32.png) [@Drew\_H\_2](https://discuss.elastic.co/u/Drew_H_2)\ **Post date:** [April 2, 2012, 2:11pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/1 "2012-04-02T14:11:32Z")
Hey guys. Just wanted to say I really like the product but I'm having an issue. Right now when I search, I'm doing this. searchRequestBuilder.setSize( 99999999 ); I've indexed data from a huge table in MySQL. I want to make sure I always get all results. This number may work now but what if it gets bigger? If I put -1 it defaults back to 10. Is there a way I can always fetch all results? Thanks for your help. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 2, 2012, 2:39pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/2 "2012-04-02T14:39:46Z")
If you need to extract docs from ES (which is slightly different than searching), then you should use the scan & scroll APIs : > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. [http://www.elasticsearch.org/guide/reference/api/search/scroll.html](http://www.elasticsearch.org/guide/reference/api/search/scroll.html) If you want to do searches, you don't have to setSize. It will return 10 docs from the whole resultset. Then, you will have only to play with the from parameter to fetch 10 other results : > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. [http://www.elasticsearch.org/guide/reference/api/search/from-size.html](http://www.elasticsearch.org/guide/reference/api/search/from-size.html) HTH David. Le 2 avril 2012 à 16:11, Drew H [drewhjava@gmail.com](mailto:drewhjava@gmail.com) a écrit : > really like the product but I'm having an issue. > > Right now when I search, I'm doing this. > > searchRequestBuilder.setSize( 99999999 ); > > I've indexed data from a huge table in MySQL. > > I want to make sure I always get all results. This number may work now but > what if it gets bigger? If I put -1 it defaults back to 10. Is there a > way I can always fetch all results? Thanks for your help. -- David Pilato [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet ---
**Author:** ![Drew\_H\_2](https://avatars.discourse-cdn.com/v4/letter/d/77aa72/32.png) [@Drew\_H\_2](https://discuss.elastic.co/u/Drew_H_2)\ **Post date:** [April 2, 2012, 3:44pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/3 "2012-04-02T15:44:04Z")
Hi David thanks for your response. In this case I don't think I can use scan & scroll. It clearly states that it's not intended for real time user requests. What I have to do is get a result set from date range as a http request. We have looked at Elasticsearch to speed up this process. Right now even with a highly optimized MySQL database, the request still takes a long time. It's just simply a lot of data and it brings down the website because of the times. Maybe Elasticsearch is not intended for this use case but it sure has greatly improved our response times in initial testing. On Monday, April 2, 2012 10:39:46 AM UTC-4, David Pilato wrote: > If you need to extract docs from ES (which is slightly different than > searching), then you should use the scan & scroll APIs : > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/scroll.html) > > If you want to do searches, you don't have to setSize. It will return 10 > docs from the whole resultset. Then, you will have only to play with the > from parameter to fetch 10 other results : > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/from-size.html) > > HTH > > David. > > Hey guys. Just wanted to say I really like the product but I'm having > an issue. > > Right now when I search, I'm doing this. > > searchRequestBuilder.setSize( 99999999 ); > > I've indexed data from a huge table in MySQL. > > I want to make sure I always get all results. This number may work now > but what if it gets bigger? If I put -1 it defaults back to 10. Is there > a way I can always fetch all results? Thanks for your help. > > -- > David Pilato > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > Twitter : @dadoonet ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 2, 2012, 4:03pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/4 "2012-04-02T16:03:53Z")
In your use case, does the user really need to fetch and display all datas on his screen ? Or do you simply need to display the more relevant docs ? But, you can create also indexes based on dates (one per month for example) and then fetch the content month per month??? Not sure I answered to your needs 😉 David. Le 2 avril 2012 à 17:44, Drew H [drewhjava@gmail.com](mailto:drewhjava@gmail.com) a écrit : > In this case I don't think I can use scan & scroll. It clearly states that > it's not intended for real time user requests. What I have to do is get a > result set from date range as a http request. We have looked at > Elasticsearch to speed up this process. Right now even with a highly > optimized MySQL database, the request still takes a long time. It's just > simply a lot of data and it brings down the website because of the times. > > Maybe Elasticsearch is not intended for this use case but it sure has > greatly improved our response times in initial testing. > > On Monday, April 2, 2012 10:39:46 AM UTC-4, David Pilato wrote: > > > If you need to extract docs from ES (which is slightly different than > > searching), then you should use the scan & scroll APIs : > > [http://www.elasticsearch.org/](http://www.elasticsearch.org/) guide/reference/api/search/ scroll.html > > [http://www.elasticsearch.org/guide/reference/api/search/scroll.html](http://www.elasticsearch.org/guide/reference/api/search/scroll.html) > > > > If you want to do searches, you don't have to setSize. It will return 10 > > docs from the whole resultset. Then, you will have only to play with the > > from parameter to fetch 10 other results : [http://www.elasticsearch.org/](http://www.elasticsearch.org/) > > guide/reference/api/search/ from-size.html > > [http://www.elasticsearch.org/guide/reference/api/search/from-size.html](http://www.elasticsearch.org/guide/reference/api/search/from-size.html) > > > > HTH > > David. > > > > > really like the product but I'm having an issue. > > > > > > Right now when I search, I'm doing this. > > > > > > searchRequestBuilder.setSize( 99999999 ); > > > > > > I've indexed data from a huge table in MySQL. > > > > > > I want to make sure I always get all results. This number may work now > > > but > > > what if it gets bigger? If I put -1 it defaults back to 10. Is there a > > > way I can always fetch all results? Thanks for your help. > > > > -- > > David Pilato > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > Twitter : @dadoonet -- David Pilato [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 2, 2012, 4:42pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/5 "2012-04-02T16:42:40Z")
To make it clear, setSize() defines how many hits in form of full- fledged documents are being transferred from Elasticsearch to the client. It has nothing to do with the index size coverage, it does not determine a search limit over this number of documents indexed. Nobody wants to transfer 99.999.999 full documents over the wire at each query. You won't be able to do that in "real time". For downloading large numbers of documents, scan/scroll is the method. Like David suggests, for querying date ranges in an inverted index, a discretization of date periods is recommended because of performance gains. Often, high resolutions like seconds or minutes are not needed. Consider you want to select documents by day ranges, then index just the day number since the beginning of your data, as an integer. Then evaluate user given dates to day numbers, as a filter to an Elasticsearch query. Integer-based filters are compact and fast. Jörg On Apr 2, 6:03 pm, "[da...@pilato.fr](mailto:da...@pilato.fr)" [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > In your use case, does the user really need to fetch and display all datas > on his screen ? > Or do you simply need to display the more relevant docs ? > > But, you can create also indexes based on dates (one per month for example) > and then fetch the content month per month??? > > Not sure I answered to your needs 😉 > > David. > > Le 2 avril 2012 à 17:44, Drew H [drewhj...@gmail.com](mailto:drewhj...@gmail.com) a écrit : > > > In this case I don't think I can use scan & scroll. It clearly states that > > it's not intended for real time user requests. What I have to do is get a > > result set from date range as a http request. We have looked at > > Elasticsearch to speed up this process. Right now even with a highly > > optimized MySQL database, the request still takes a long time. It's just > > simply a lot of data and it brings down the website because of the times. > > > Maybe Elasticsearch is not intended for this use case but it sure has > > greatly improved our response times in initial testing. > > > On Monday, April 2, 2012 10:39:46 AM UTC-4, David Pilato wrote: > > > > > If you need to extract docs from ES (which is slightly different than > > > searching), then you should use the scan & scroll APIs : > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/) scroll.html > > > [http://www.elasticsearch.org/guide/reference/api/search/scroll.html](http://www.elasticsearch.org/guide/reference/api/search/scroll.html) > > > > If you want to do searches, you don't have to setSize. It will return 10 > > > docs from the whole resultset. Then, you will have only to play with the > > > from parameter to fetch 10 other results : [http://www.elasticsearch.org/](http://www.elasticsearch.org/) > > > guide/reference/api/search/ from-size.html > > > [http://www.elasticsearch.org/guide/reference/api/search/from-size.html](http://www.elasticsearch.org/guide/reference/api/search/from-size.html) > > > > HTH > > > David. > > > > > really like the product but I'm having an issue. > > > > > Right now when I search, I'm doing this. > > > > > searchRequestBuilder.setSize( 99999999 ); > > > > > I've indexed data from a huge table in MySQL. > > > > > I want to make sure I always get all results. This number may work now > > > > but > > > > what if it gets bigger? If I put -1 it defaults back to 10. Is there a > > > > way I can always fetch all results? Thanks for your help. > > > > -- > > > David Pilato > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > Twitter : @dadoonet > > -- > David Pilatohttp://dev.david.pilato.fr/ > Twitter : @dadoonet ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 2, 2012, 4:52pm UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/6 "2012-04-02T16:52:24Z")
Thanks Jörg ! I understand now what I was trying to say 😉 Cheers, David. > -----Message d'origine----- > De : [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) > [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] De la part de Jörg Prante > Envoyé : lundi 2 avril 2012 18:43 > À : elasticsearch > Objet : Re: Always view all results from search? (Java Client) > > To make it clear, setSize() defines how many hits in form of full- > fledged documents are being transferred from Elasticsearch to the > client. > > It has nothing to do with the index size coverage, it does not > determine a search limit over this number of documents indexed. > > Nobody wants to transfer 99.999.999 full documents over the wire at > each query. You won't be able to do that in "real time". For > downloading large numbers of documents, scan/scroll is the method. > > Like David suggests, for querying date ranges in an inverted index, a > discretization of date periods is recommended because of performance > gains. Often, high resolutions like seconds or minutes are not needed. > Consider you want to select documents by day ranges, then index just > the day number since the beginning of your data, as an integer. Then > evaluate user given dates to day numbers, as a filter to an > Elasticsearch query. Integer-based filters are compact and fast. > > Jörg > > On Apr 2, 6:03 pm, "[da...@pilato.fr](mailto:da...@pilato.fr)" [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > In your use case, does the user really need to fetch and display all > > datas on his screen ? > > Or do you simply need to display the more relevant docs ? > > > > But, you can create also indexes based on dates (one per month for > > example) and then fetch the content month per month??? > > > > Not sure I answered to your needs 😉 > > > > David. > > > > Le 2 avril 2012 à 17:44, Drew H [drewhj...@gmail.com](mailto:drewhj...@gmail.com) a écrit : > > > > > In this case I don't think I can use scan & scroll. It clearly > > > states that it's not intended for real time user requests. What I > > > have to do is get a result set from date range as a http request. > > > We have looked at Elasticsearch to speed up this process. Right > > > now > > > even with a highly optimized MySQL database, the request still > > > takes > > > a long time. It's just simply a lot of data and it brings down the > > > website because of the times. > > > > > Maybe Elasticsearch is not intended for this use case but it sure > > > has greatly improved our response times in initial testing. > > > > > On Monday, April 2, 2012 10:39:46 AM UTC-4, David Pilato wrote: > > > > > > > If you need to extract docs from ES (which is slightly different > > > > than searching), then you should use the scan & scroll APIs : > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/) > > > > scroll.html > > > > \<[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/scroll.htm) > > > > > > l\> > > > > > > If you want to do searches, you don't have to setSize. It will > > > > return 10 docs from the whole resultset. Then, you will have only > > > > to play with the from parameter to fetch 10 other results : > > > > [http://www.elasticsearch.org/](http://www.elasticsearch.org/) guide/reference/api/search/ > > > > from-size.html > > > > \<[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/from-) > > > > size > > > > .html\> > > > > > > HTH > > > > David. > > > > > > > really like the product but I'm having an issue. > > > > > > > Right now when I search, I'm doing this. > > > > > > > searchRequestBuilder.setSize( 99999999 ); > > > > > > > I've indexed data from a huge table in MySQL. > > > > > > > I want to make sure I always get all results. This number may > > > > > work now but what if it gets bigger? If I put -1 it defaults > > > > > back to 10. Is there a way I can always fetch all results? > > > > > Thanks for your help. > > > > > > -- > > > > David Pilato > > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > > Twitter : @dadoonet > > > > -- > > David Pilatohttp://dev.david.pilato.fr/ Twitter : @dadoonet ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:33am UTC](https://discuss.elastic.co/t/always-view-all-results-from-search-java-client/7202/7 "2017-07-06T03:33:57Z")