# Intermittently search slow response **URL:** \ **Category:** Elasticsearch\ **Created:** [November 29, 2013, 10:51am UTC](https://discuss.elastic.co/t/intermittently-search-slow-response/14644 "2013-11-29T10:51:35Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [November 29, 2013, 10:51am UTC](https://discuss.elastic.co/t/intermittently-search-slow-response/14644/1 "2013-11-29T10:51:35Z")
Hi, we are experience a strange issue, it happen intermittently. we have three elasticsearch nodes running version 0.90.7 and started a new cluster. Then as data coming in, we query one of the nodes, sometime the result return immediately. Sometime it just hang there forever until ctrl+z is issued. Not exactly sure what is wrong. Can anyone pinpoint where might be the possible cause? [user@node1 ~]$ time curl -XGET '[http://localhost:9200/index\_test/\_search?q=key:keywords&pretty=true](http://localhost:9200/index_test/_search?q=key:keywords&pretty=true)' { "took" : 69, "timed\_out" : false, "\_shards" : { "total" : 4500, "successful" : 4500, "failed" : 0 }, "hits" : { "total" : 2201, "max\_score" : 6.192957, "hits" : [ { "\_index" : "index\_test", ... ... ... real 0m0.072s user 0m0.001s sys 0m0.003s [user@node1 ~]$ time curl -XGET '[http://localhost:9200/index\_test/\_search?q=key:keywords&pretty=true](http://localhost:9200/index_test/_search?q=key:keywords&pretty=true)' ^C real 16m34.575s user 0m0.006s sys 0m0.012s -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/868a0a1f-ee19-4a2d-807c-1379f023e9f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/868a0a1f-ee19-4a2d-807c-1379f023e9f2%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [November 29, 2013, 9:38pm UTC](https://discuss.elastic.co/t/intermittently-search-slow-response/14644/2 "2013-11-29T21:38:03Z")
you are searching on 4500 shards? Is that right? I am just asking there could be a lot of stuff going on though. simon On Friday, November 29, 2013 11:51:35 AM UTC+1, Jason Wee wrote: > Hi, we are experience a strange issue, it happen intermittently. we have > three elasticsearch nodes running version 0.90.7 and started a new cluster. > Then as data coming in, we query one of the nodes, sometime the result > return immediately. Sometime it just hang there forever until ctrl+z is > issued. Not exactly sure what is wrong. Can anyone pinpoint where might be > the possible cause? > > [user@node1 ~]$ time curl -XGET ' > [http://localhost:9200/index\_test/\_search?q=key:keywords&pretty=true](http://localhost:9200/index_test/_search?q=key:keywords&pretty=true)' > { > "took" : 69, > "timed\_out" : false, > "\_shards" : { > "total" : 4500, > "successful" : 4500, > "failed" : 0 > }, > "hits" : { > "total" : 2201, > "max\_score" : 6.192957, > "hits" : [ { > "\_index" : "index\_test", > ... > ... > ... > real 0m0.072s > user 0m0.001s > sys 0m0.003s > > [user@node1 ~]$ time curl -XGET ' > [http://localhost:9200/index\_test/\_search?q=key:keywords&pretty=true](http://localhost:9200/index_test/_search?q=key:keywords&pretty=true)' > > ^C > > real 16m34.575s > user 0m0.006s > sys 0m0.012s -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 29, 2013, 9:52pm UTC](https://discuss.elastic.co/t/intermittently-search-slow-response/14644/3 "2013-11-29T21:52:38Z")
4500 shards isn't too bad, we've got 5500 and queries are quick. You haven't given us much of a clue on what size cluster you have, including node size, RAM usage, replica count etc, which will impact query performance. Try installing the elastichq plugin, it has some nice functionality under Node Diagnostics that lets you see where things might be getting out of hand. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 30 November 2013 08:38, simonw [simon.willnauer@elasticsearch.com](mailto:simon.willnauer@elasticsearch.com) wrote: > you are searching on 4500 shards? Is that right? I am just asking there > could be a lot of stuff going on though. > > simon > On Friday, November 29, 2013 11:51:35 AM UTC+1, Jason Wee wrote: > > > Hi, we are experience a strange issue, it happen intermittently. we have > > three elasticsearch nodes running version 0.90.7 and started a new cluster. > > Then as data coming in, we query one of the nodes, sometime the result > > return immediately. Sometime it just hang there forever until ctrl+z is > > issued. Not exactly sure what is wrong. Can anyone pinpoint where might be > > the possible cause? > > > > [user@node1 ~]$ time curl -XGET '[http://localhost:9200/index\_](http://localhost:9200/index_) > > test/\_search?q=key:keywords&pretty=true' > > { > > "took" : 69, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4500, > > "successful" : 4500, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 2201, > > "max\_score" : 6.192957, > > "hits" : [ { > > "\_index" : "index\_test", > > ... > > ... > > ... > > real 0m0.072s > > user 0m0.001s > > sys 0m0.003s > > > > [user@node1 ~]$ time curl -XGET '[http://localhost:9200/index\_](http://localhost:9200/index_) > > test/\_search?q=key:keywords&pretty=true' > > > > ^C > > > > real 16m34.575s > > user 0m0.006s > > sys 0m0.012s > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624agH6H2q-8YuknGzZH\_7wWhVB1p9u8kscRRUZO5EOm%2Bzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624agH6H2q-8YuknGzZH_7wWhVB1p9u8kscRRUZO5EOm%2Bzg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [November 30, 2013, 5:22pm UTC](https://discuss.elastic.co/t/intermittently-search-slow-response/14644/4 "2013-11-30T17:22:35Z")
Yes, the index is started with a total of 4500 shards. We are currently have three dedicated server running for Elasticsearch cluster. The three nodes has identical hardware configuration. Each node is running intel i7-4770 cpu with 32GB of ram. Each node is allocated 16GB heap out of 32GB of total memory and it only run elasticsearch and no other jvm or intensive cpu processes. Our es cluster currently having 2 replicas and total active shards currently is 15000. when curl on [http://localhost:9200/\_cluster/nodes/node1/stats?all](http://localhost:9200/_cluster/nodes/node1/stats?all) indices store is around 11GB at this moment. Heap usage hovering in between 6G and 13G. Non heap usage is low at around 60MB. Jvm threads count stay constant around 120. when curl on [http://localhost:9200/\_stats](http://localhost:9200/_stats) , the path to index current, indices/index\_test/total/indexing/index\_current hovering around 240 constant throughout. Will these information give you any clue what might happen to the intermittently search slow response? =\> Try installing the elastichq plugin, it has some nice functionality under Node Diagnostics that lets you see where things might be getting out of hand. Thank you, will install this. /Jason On Sat, Nov 30, 2013 at 5:52 AM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > 4500 shards isn't too bad, we've got 5500 and queries are quick. > > You haven't given us much of a clue on what size cluster you have, > including node size, RAM usage, replica count etc, which will impact query > performance. Try installing the elastichq plugin, it has some nice > functionality under Node Diagnostics that lets you see where things might > be getting out of hand. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 30 November 2013 08:38, simonw [simon.willnauer@elasticsearch.com](mailto:simon.willnauer@elasticsearch.com)wrote: > > > you are searching on 4500 shards? Is that right? I am just asking there > > could be a lot of stuff going on though. > > > > simon > > On Friday, November 29, 2013 11:51:35 AM UTC+1, Jason Wee wrote: > > > > > Hi, we are experience a strange issue, it happen intermittently. we have > > > three elasticsearch nodes running version 0.90.7 and started a new cluster. > > > Then as data coming in, we query one of the nodes, sometime the result > > > return immediately. Sometime it just hang there forever until ctrl+z is > > > issued. Not exactly sure what is wrong. Can anyone pinpoint where might be > > > the possible cause? > > > > > > [user@node1 ~]$ time curl -XGET '[http://localhost:9200/index\_](http://localhost:9200/index_) > > > test/\_search?q=key:keywords&pretty=true' > > > { > > > "took" : 69, > > > "timed\_out" : false, > > > "\_shards" : { > > > "total" : 4500, > > > "successful" : 4500, > > > "failed" : 0 > > > }, > > > "hits" : { > > > "total" : 2201, > > > "max\_score" : 6.192957, > > > "hits" : [ { > > > "\_index" : "index\_test", > > > ... > > > ... > > > ... > > > real 0m0.072s > > > user 0m0.001s > > > sys 0m0.003s > > > > > > [user@node1 ~]$ time curl -XGET '[http://localhost:9200/index\_](http://localhost:9200/index_) > > > test/\_search?q=key:keywords&pretty=true' > > > > > > ^C > > > > > > real 16m34.575s > > > user 0m0.006s > > > sys 0m0.012s > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f7a0c71d-d54e-4be8-a487-47371ca61456%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEM624agH6H2q-8YuknGzZH\_7wWhVB1p9u8kscRRUZO5EOm%2Bzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624agH6H2q-8YuknGzZH_7wWhVB1p9u8kscRRUZO5EOm%2Bzg%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4ityaBRoGoOf9p3GcUNy75eiXtA9F5KK24YwXuLzRZ43txw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4ityaBRoGoOf9p3GcUNy75eiXtA9F5KK24YwXuLzRZ43txw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:04am UTC](https://discuss.elastic.co/t/intermittently-search-slow-response/14644/5 "2017-07-06T02:04:04Z")