# Elasticsearch timeout for search query **URL:** \ **Category:** Elasticsearch\ **Created:** [April 28, 2020, 3:06am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067 "2020-04-28T03:06:45Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\ **Post date:** [April 28, 2020, 3:06am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/1 "2020-04-28T03:06:46Z")
Hi, `Elasticsearch version 7.6.1` `We have a total of 55 indices with 228 shards and a disk space of 4.8 TB` Our indexing rate ranges from (`3000-8000 docs per second`) with a total of 250 million docs coming per day. We have `10 data nodes` running each with 2 cores and 5 GB of RAM (50% heap) capacity and we have a total of around 4.5 billion documents at the moment: Here's out heap usage in the past 1 week (max heap is 25 GB): ![Screen Shot 2020-04-27 at 1.21.33 PM](https://us1.discourse-cdn.com/elastic/original/3X/4/f/4fd96c0b6048464ce70ed1761ad629de93e3df6d.png) When I run `a query to get all the data` (4.8 billion docs), the query passes sometimes and fails sometimes irrespective of the amount of indexing happening at that time. `CPU utilization on all the nodes reach 100% when the query is running` No-one is running queries on the cluster except for me. search thread pool queue count doesn't cross 60 when the query is running: ![Screen Shot 2020-04-27 at 7.52.57 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/3/3398eed00e493198562e6d24bed7555c71a92c80.png) Even though the official docs insist on setting the heap to 50% of available RAM, it seems like we are not using most of the heap available, do you think the search would improve if I decrease the heap size to 1.5GB? OR is there another way to improve search performance? - Thank you ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 28, 2020, 5:15am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/2 "2020-04-28T05:15:46Z")
What type of query do you run to get all data? How does it fail? Have you looked at disk utilization and iowait while you are running the query? ---
**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\ **Post date:** [April 28, 2020, 6:00am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/3 "2020-04-28T06:00:45Z")
Thank you for a quick reply - I run the query in kibana UI like this to get all the data from the last 25 days - It fails in kibana with the following error message (not sure if this matters because sometimes the query passes): ```auto SyntaxError: Unexpected token < in JSON at position 0 at JSON.parse () at https://elk.company.com/bundles/commons.bundle.js:3:3380253 at https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:94842 at https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:94980 at u.$digest (https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:100155) at u.$apply (https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:102334) at https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:65714 at w (https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:68618) at XMLHttpRequest.b.onload (https://elk.company.com/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:67981) ``` - disk utilization is not at all reaching the peak, we use 10 EBS volumes each with 1000GB which means we have 3000IOPS limit and when I just ran the query, the read IOPS didn't even reach a sum of 100 - here's the IOwait graph for the underlying EC2 instances (m5.4xlarge - 16 cores with 64 GB of RAM) that host the Elasticsearch data nodes ---
**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\ **Post date:** [April 30, 2020, 12:19am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/4 "2020-04-30T00:19:52Z")
any update here please? - Thank you ---
**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\ **Post date:** [May 1, 2020, 12:47am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/5 "2020-05-01T00:47:15Z")
Also in most of our nodes JVM usage looks like "saw-tooth", AFAIK this is over-allocation of RAM for JVM...right? ![Screen Shot 2020-04-30 at 5.45.31 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/1/d189dca5c3140e1027d29aa833518bd837887530.png) ---
**Author:** ![Rakesh\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rakesh_b/32/48128_2.png) [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\ **Post date:** [May 2, 2020, 12:47am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/6 "2020-05-02T00:47:09Z")
I feel like this cannot be an IO issue, we just re-built the cluster and the nodes are able to read like `1GB per second some times`, I mean to say that the nodes are not utilizing the full potential of the EBS volumes when I ran a query to get the data from the last 30 days ![Screen Shot 2020-05-01 at 5.45.48 PM](https://us1.discourse-cdn.com/elastic/original/3X/7/0/70b747ca84ae9e938ba0e0f87fbee1cbdfb67924.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 2, 2020, 7:23am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/7 "2020-05-02T07:23:48Z")
That saw tooth pattern is what you are looking for and very healthy. You look to have a reasonable heap size. I am not sure about the Kibana error though. Can you capture the query being sent and try running this from the dev console in Kibana? Enabling the slow log and looking at what this shows might also be a good idea. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 30, 2020, 7:23am UTC](https://discuss.elastic.co/t/elasticsearch-timeout-for-search-query/230067/8 "2020-05-30T07:23:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.