# Queue size **URL:** \ **Category:** Elasticsearch\ **Created:** [February 20, 2015, 4:32pm UTC](https://discuss.elastic.co/t/queue-size/22297 "2015-02-20T16:32:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Christopher\_Bourez](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@Christopher\_Bourez](https://discuss.elastic.co/u/Christopher_Bourez)\ **Post date:** [February 20, 2015, 4:32pm UTC](https://discuss.elastic.co/t/queue-size/22297/1 "2015-02-20T16:32:53Z")
I'm having the following problem : "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt\_prod][3]: EsRejectedExecutionException[rejected execution (queue capacity 1000) on org.elasticsearch.action.search.type.TransportSearchTypeAction It sounds very strange; when I restarted the server, it worked fine again. What could happen ? Here is my configuration : - ES version is 1.0.1 - I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one has 5 shards - the cluster is one only one instance (solo) - the state of the cluster says 15 successful shard, 0 failed shard and 30 total shards (where are the 15 shards missing ?) - in my settings, mlockall is set to true - I enabled script.disable\_dynamic: false, installed plugins \_head and action-updatebyquery - ES heap size is correctly set to 50% by the recipe which I can confirm using top command : 5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java - I'm using only 30% of disk capacity My traffic is not more than 125 requests per minutes : [https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png](https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png) So if I understand well, each request can live 30s, how come I have a queue of 1000 ?! Can ES save the requests in the queue while the shards have failed ? Why do the shards do not come back ? Thanks for your help (I'm not using ES usually, more Solr or CloudSearch) (I also posted it here : [https://github.com/elasticsearch/elasticsearch/issues/9792](https://github.com/elasticsearch/elasticsearch/issues/9792) ) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 20, 2015, 9:04pm UTC](https://discuss.elastic.co/t/queue-size/22297/2 "2015-02-20T21:04:06Z")
It means your cluster is probably overloaded. Your missing shards are probably replicas, which will never be assigned with a one node cluster, head should show these. What is that massive spike right before the end of the graph? Are you also monitoring things like load and other OS level stats? On 21 February 2015 at 03:32, Christopher Bourez \< [christopher.bourez@gmail.com](mailto:christopher.bourez@gmail.com)\> wrote: > I'm having the following problem : > > "SearchPhaseExecutionException[Failed to execute phase [query], all shards > failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt\_prod][3]: > EsRejectedExecutionException[rejected execution (queue capacity 1000) on > org.elasticsearch.action.search.type.TransportSearchTypeAction > > It sounds very strange; when I restarted the server, it worked fine again. > > What could happen ? > > Here is my configuration : > > - ES version is 1.0.1 > - I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one has 5 > shards > - the cluster is one only one instance (solo) > - the state of the cluster says 15 successful shard, 0 failed shard and 30 > total shards (where are the 15 shards missing ?) > - in my settings, mlockall is set to true > - I enabled script.disable\_dynamic: false, installed plugins \_head and > action-updatebyquery > - ES heap size is correctly set to 50% by the recipe which I can confirm > using top command : > 5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java > - I'm using only 30% of disk capacity > > My traffic is not more than 125 requests per minutes : > > [https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png](https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png) > > So if I understand well, each request can live 30s, how come I have a > queue of 1000 ?! > Can ES save the requests in the queue while the shards have failed ? > Why do the shards do not come back ? > > Thanks for your help (I'm not using ES usually, more Solr or CloudSearch) > > (I also posted it here : > [Queue size · Issue #9792 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/9792) ) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 22, 2015, 8:05pm UTC](https://discuss.elastic.co/t/queue-size/22297/3 "2015-02-22T20:05:00Z")
I assume search actions got stuck and block the subsequent ones, which results in the search queue filling up. Maybe the cause is printed in the server logs. Setting replica to 0 with just one node helps to fix the 15 shards/30 total shards count but that is an unrelated story. Jörg On Fri, Feb 20, 2015 at 10:04 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > It means your cluster is probably overloaded. > Your missing shards are probably replicas, which will never be assigned > with a one node cluster, head should show these. > > What is that massive spike right before the end of the graph? Are you also > monitoring things like load and other OS level stats? > > On 21 February 2015 at 03:32, Christopher Bourez \< > [christopher.bourez@gmail.com](mailto:christopher.bourez@gmail.com)\> wrote: > > > I'm having the following problem : > > > > "SearchPhaseExecutionException[Failed to execute phase [query], all > > shards failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt\_prod][3]: > > EsRejectedExecutionException[rejected execution (queue capacity 1000) on > > org.elasticsearch.action.search.type.TransportSearchTypeAction > > > > It sounds very strange; when I restarted the server, it worked fine again. > > > > What could happen ? > > > > Here is my configuration : > > > > - ES version is 1.0.1 > > - I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one has > > 5 shards > > - the cluster is one only one instance (solo) > > - the state of the cluster says 15 successful shard, 0 failed shard and > > 30 total shards (where are the 15 shards missing ?) > > - in my settings, mlockall is set to true > > - I enabled script.disable\_dynamic: false, installed plugins \_head and > > action-updatebyquery > > - ES heap size is correctly set to 50% by the recipe which I can confirm > > using top command : > > 5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java > > - I'm using only 30% of disk capacity > > > > My traffic is not more than 125 requests per minutes : > > > > [https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png](https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png) > > > > So if I understand well, each request can live 30s, how come I have a > > queue of 1000 ?! > > Can ES save the requests in the queue while the shards have failed ? > > Why do the shards do not come back ? > > > > Thanks for your help (I'm not using ES usually, more Solr or CloudSearch) > > > > (I also posted it here : > > [Queue size · Issue #9792 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/9792) ) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHse40C\_-hUfQynF2c\_mjWgPtSZdssgtea8-zKbWEv0Dw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHse40C_-hUfQynF2c_mjWgPtSZdssgtea8-zKbWEv0Dw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Christopher\_Bourez](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@Christopher\_Bourez](https://discuss.elastic.co/u/Christopher_Bourez)\ **Post date:** [February 23, 2015, 10:48am UTC](https://discuss.elastic.co/t/queue-size/22297/4 "2015-02-23T10:48:01Z")
Ok for default replica number in index settings... sounds good index\_indexing\_slowlog and index\_indexing\_slowlog are void but errors appear in the main logs. let me find them next time But is there any way to put a timeout in the server on queries ? because I thought they would not last more than 30s and having a maximum of 150 requests per minutes should not fill the queue at any time. On Sunday, February 22, 2015 at 9:05:08 PM UTC+1, Jörg Prante wrote: > I assume search actions got stuck and block the subsequent ones, which > results in the search queue filling up. Maybe the cause is printed in the > server logs. > > Setting replica to 0 with just one node helps to fix the 15 shards/30 > total shards count but that is an unrelated story. > > Jörg > > On Fri, Feb 20, 2015 at 10:04 PM, Mark Walkom \<[markw...@gmail.com](mailto:markw...@gmail.com) > \\> wrote: > > > It means your cluster is probably overloaded. > > Your missing shards are probably replicas, which will never be assigned > > with a one node cluster, head should show these. > > > > What is that massive spike right before the end of the graph? Are you > > also monitoring things like load and other OS level stats? > > > > On 21 February 2015 at 03:32, Christopher Bourez \<[christoph...@gmail.com](mailto:christoph...@gmail.com) > > \\> wrote: > > > > > I'm having the following problem : > > > > > > "SearchPhaseExecutionException[Failed to execute phase [query], all > > > shards failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt\_prod][3]: > > > EsRejectedExecutionException[rejected execution (queue capacity 1000) on > > > org.elasticsearch.action.search.type.TransportSearchTypeAction > > > > > > It sounds very strange; when I restarted the server, it worked fine > > > again. > > > > > > What could happen ? > > > > > > Here is my configuration : > > > > > > - ES version is 1.0.1 > > > - I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one has > > > 5 shards > > > - the cluster is one only one instance (solo) > > > - the state of the cluster says 15 successful shard, 0 failed shard and > > > 30 total shards (where are the 15 shards missing ?) > > > - in my settings, mlockall is set to true > > > - I enabled script.disable\_dynamic: false, installed plugins \_head and > > > action-updatebyquery > > > - ES heap size is correctly set to 50% by the recipe which I can confirm > > > using top command : > > > 5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java > > > - I'm using only 30% of disk capacity > > > > > > My traffic is not more than 125 requests per minutes : > > > > > > [https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png](https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png) > > > > > > So if I understand well, each request can live 30s, how come I have a > > > queue of 1000 ?! > > > Can ES save the requests in the queue while the shards have failed ? > > > Why do the shards do not come back ? > > > > > > Thanks for your help (I'm not using ES usually, more Solr or CloudSearch) > > > > > > (I also posted it here : > > > [Queue size · Issue #9792 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/9792) ) > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 23, 2015, 4:25pm UTC](https://discuss.elastic.co/t/queue-size/22297/5 "2015-02-23T16:25:50Z")
Once a query is submitted, Elasticsearch will execute the query until it terminates. A query timeout only returns results prematurely, it does not cancel ongoing query threads on nodes. Jörg On Mon, Feb 23, 2015 at 11:48 AM, Christopher Bourez \< [christopher.bourez@gmail.com](mailto:christopher.bourez@gmail.com)\> wrote: > Ok for default replica number in index settings... sounds good > > index\_indexing\_slowlog and index\_indexing\_slowlog are void but errors > appear in the main logs. let me find them next time > > But is there any way to put a timeout in the server on queries ? because I > thought they would not last more than 30s and having a maximum of 150 > requests per minutes should not fill the queue at any time. > > On Sunday, February 22, 2015 at 9:05:08 PM UTC+1, Jörg Prante wrote: > > > I assume search actions got stuck and block the subsequent ones, which > > results in the search queue filling up. Maybe the cause is printed in the > > server logs. > > > > Setting replica to 0 with just one node helps to fix the 15 shards/30 > > total shards count but that is an unrelated story. > > > > Jörg > > > > On Fri, Feb 20, 2015 at 10:04 PM, Mark Walkom [markw...@gmail.com](mailto:markw...@gmail.com) wrote: > > > > > It means your cluster is probably overloaded. > > > Your missing shards are probably replicas, which will never be assigned > > > with a one node cluster, head should show these. > > > > > > What is that massive spike right before the end of the graph? Are you > > > also monitoring things like load and other OS level stats? > > > > > > On 21 February 2015 at 03:32, Christopher Bourez \<[christoph...@gmail.com](mailto:christoph...@gmail.com) > > > > > > > wrote: > > > > > > > I'm having the following problem : > > > > > > > > "SearchPhaseExecutionException[Failed to execute phase [query], all > > > > shards failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt\_prod][3]: > > > > EsRejectedExecutionException[rejected execution (queue capacity 1000) > > > > on org.elasticsearch.action.search.type.TransportSearchTypeAction > > > > > > > > It sounds very strange; when I restarted the server, it worked fine > > > > again. > > > > > > > > What could happen ? > > > > > > > > Here is my configuration : > > > > > > > > - ES version is 1.0.1 > > > > - I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one > > > > has 5 shards > > > > - the cluster is one only one instance (solo) > > > > - the state of the cluster says 15 successful shard, 0 failed shard and > > > > 30 total shards (where are the 15 shards missing ?) > > > > - in my settings, mlockall is set to true > > > > - I enabled script.disable\_dynamic: false, installed plugins \_head and > > > > action-updatebyquery > > > > - ES heap size is correctly set to 50% by the recipe which I can > > > > confirm using top command : > > > > 5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java > > > > - I'm using only 30% of disk capacity > > > > > > > > My traffic is not more than 125 requests per minutes : > > > > > > > > [https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png](https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png) > > > > > > > > So if I understand well, each request can live 30s, how come I have a > > > > queue of 1000 ?! > > > > Can ES save the requests in the queue while the shards have failed ? > > > > Why do the shards do not come back ? > > > > > > > > Thanks for your help (I'm not using ES usually, more Solr or > > > > CloudSearch) > > > > > > > > (I also posted it here : [https://github.com/](https://github.com/) > > > > elasticsearch/elasticsearch/issues/9792 ) > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg > > > %[40mail.gmail.com](http://40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQOADao45rfdFngVpYWKy\_XDU3HAyRci2jbP2ugf2BEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQOADao45rfdFngVpYWKy_XDU3HAyRci2jbP2ugf2BEg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Christopher\_Bourez](https://avatars.discourse-cdn.com/v4/letter/c/b19c9b/32.png) [@Christopher\_Bourez](https://discuss.elastic.co/u/Christopher_Bourez)\ **Post date:** [February 25, 2015, 3:38pm UTC](https://discuss.elastic.co/t/queue-size/22297/6 "2015-02-25T15:38:51Z")
OK, not very great that there is no timeout. It is indeed the search queue that goes to 1000 /\_cat/thread\_pool?v&h=id,host,suggest.active,suggest.rejected,suggest.completed,search.queue It is a request that breaks all the shards (no replica), because I only find one error in the log file (others are "lastShard error"). Not very great from ES to be broken by one request ?! The error says : 2015-02-25 00:07:44,973][DEBUG][action.search.type] > [production21-ebs2.localdomain] [231571] Failed to execute fetch phase > > [Error: Runtime.getRuntime().exec("/tmp/cnmm").getInputStream(): Cannot > run program "/tmp/cnmm": error=13, Permission denied] > > [Near : {... w InputStreamReader(Runtime.getRuntime().exec("/tm ....}] > > It > sounds [Elasticsearch server stops due to java.io.IOException break - Stack Overflow](http://stackoverflow.com/questions/26652187/elasticsearch-server-stops-due-to-java-io-ioexception-break) I will desactivate dynamic scripting and we'll keep you in touch. I'm quite surprised by the conception of ES, or is ES too young for production ? On Monday, February 23, 2015 at 5:25:57 PM UTC+1, Jörg Prante wrote: > Once a query is submitted, Elasticsearch will execute the query until it > terminates. A query timeout only returns results prematurely, it does not > cancel ongoing query threads on nodes. > > Jörg > > On Mon, Feb 23, 2015 at 11:48 AM, Christopher Bourez \< > [christoph...@gmail.com](mailto:christoph...@gmail.com) \\> wrote: > > > Ok for default replica number in index settings... sounds good > > > > index\_indexing\_slowlog and index\_indexing\_slowlog are void but errors > > appear in the main logs. let me find them next time > > > > But is there any way to put a timeout in the server on queries ? because > > I thought they would not last more than 30s and having a maximum of 150 > > requests per minutes should not fill the queue at any time. > > > > On Sunday, February 22, 2015 at 9:05:08 PM UTC+1, Jörg Prante wrote: > > > > > I assume search actions got stuck and block the subsequent ones, which > > > results in the search queue filling up. Maybe the cause is printed in the > > > server logs. > > > > > > Setting replica to 0 with just one node helps to fix the 15 shards/30 > > > total shards count but that is an unrelated story. > > > > > > Jörg > > > > > > On Fri, Feb 20, 2015 at 10:04 PM, Mark Walkom [markw...@gmail.com](mailto:markw...@gmail.com) > > > wrote: > > > > > > > It means your cluster is probably overloaded. > > > > Your missing shards are probably replicas, which will never be assigned > > > > with a one node cluster, head should show these. > > > > > > > > What is that massive spike right before the end of the graph? Are you > > > > also monitoring things like load and other OS level stats? > > > > > > > > On 21 February 2015 at 03:32, Christopher Bourez \< > > > > [christoph...@gmail.com](mailto:christoph...@gmail.com)\> wrote: > > > > > > > > > I'm having the following problem : > > > > > > > > > > "SearchPhaseExecutionException[Failed to execute phase [query], all > > > > > shards failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt\_prod][3]: > > > > > EsRejectedExecutionException[rejected execution (queue capacity 1000) > > > > > on org.elasticsearch.action.search.type.TransportSearchTypeAction > > > > > > > > > > It sounds very strange; when I restarted the server, it worked fine > > > > > again. > > > > > > > > > > What could happen ? > > > > > > > > > > Here is my configuration : > > > > > > > > > > - ES version is 1.0.1 > > > > > - I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one > > > > > has 5 shards > > > > > - the cluster is one only one instance (solo) > > > > > - the state of the cluster says 15 successful shard, 0 failed shard > > > > > and 30 total shards (where are the 15 shards missing ?) > > > > > - in my settings, mlockall is set to true > > > > > - I enabled script.disable\_dynamic: false, installed plugins \_head and > > > > > action-updatebyquery > > > > > - ES heap size is correctly set to 50% by the recipe which I can > > > > > confirm using top command : > > > > > 5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java > > > > > - I'm using only 30% of disk capacity > > > > > > > > > > My traffic is not more than 125 requests per minutes : > > > > > > > > > > [https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png](https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png) > > > > > > > > > > So if I understand well, each request can live 30s, how come I have a > > > > > queue of 1000 ?! > > > > > Can ES save the requests in the queue while the shards have failed ? > > > > > Why do the shards do not come back ? > > > > > > > > > > Thanks for your help (I'm not using ES usually, more Solr or > > > > > CloudSearch) > > > > > > > > > > (I also posted it here : [https://github.com/](https://github.com/) > > > > > elasticsearch/elasticsearch/issues/9792 ) > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg > > > > %[40mail.gmail.com](http://40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t\_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/673a4008-46df-43b2-84df-ceb3d52474cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/673a4008-46df-43b2-84df-ceb3d52474cc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:30am UTC](https://discuss.elastic.co/t/queue-size/22297/7 "2017-07-06T00:30:10Z")