# Getting 429 ES Exception while performing search queries **URL:** \ **Category:** Elasticsearch\ **Created:** [February 4, 2019, 11:30am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945 "2019-02-04T11:30:09Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Shibbu11](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shibbu11](https://discuss.elastic.co/u/Shibbu11)\ **Post date:** [February 4, 2019, 11:30am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/1 "2019-02-04T11:30:09Z")
Hello, I am using Elasticsearch 5.6.7. I have one node in my cluster with 128 GB ram and 16 GB heap size. ![node_details](https://us1.discourse-cdn.com/elastic/original/3X/2/5/25c8d9acec3229a6219509cded4e92a020a53c37.png) I have around 14 indices with 5 primary and 1 replica shard each in this node. Each index has a size of 40-50 GB. These indices are used to store data for a particular day and we have search queries(mainly term aggregations) which allow querying from the current date to 2 weeks back. When querying for 2 weeks long data, we get this exception --- ``` { "type":"es_rejected_execution_exception", "reason":"rejected execution of [org.elasticsearch.transport.TransportService$7@6f7799fa](mailto:org.elasticsearch.transport.TransportService$7@6f7799fa) on EsThreadPoolExecutor[ **search** , **queue capacity = 1000** , [org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6c0d64d2[**Running**](mailto:org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6c0d64d2[Running) **, pool size = 25, active threads = 25, queued tasks = 3410, completed tasks = 1538807]]** " } ``` Que.1: Why are we getting so many tasks? Que.2: Will Increasing queue size be helpful? Que.3: What configuration changes may help if we can't reduce the number of requests to ES? Thanks in advance!! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 4, 2019, 11:38am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/2 "2019-02-04T11:38:30Z")
How many clients do you have concurrently querying the node? Are they sending a single query at a time? How long does these queries take to complete? ---
**Author:** ![Shibbu11](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shibbu11](https://discuss.elastic.co/u/Shibbu11)\ **Post date:** [February 4, 2019, 11:53am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/3 "2019-02-04T11:53:39Z")
There are multiple queries simultaneously sent by multiple clients. When querying for less than 1 week data queries take about 60-80 seconds. For 2 weeks data, we hit this exception. So we don't know the response time. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 4, 2019, 11:54am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/4 "2019-02-04T11:54:53Z")
What type of storage do you have? SSDs? What does CPU usage and disk I/O and iowait look like? ---
**Author:** ![Shibbu11](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shibbu11](https://discuss.elastic.co/u/Shibbu11)\ **Post date:** [February 5, 2019, 6:27am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/5 "2019-02-05T06:27:03Z")
We are using HDDs. CPU usage and disk I/O I can pass on when I am able to reproduce it in one or two days. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 5, 2019, 8:23am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/6 "2019-02-05T08:23:24Z")
If you have slow storage and queries are piling on, queues tend to fill up. I would not be surprised if you saw dramatic improvement if you switched to SSDs as these tend to handle random disk I/O a lot better than HDDs. ---
**Author:** ![Shibbu11](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shibbu11](https://discuss.elastic.co/u/Shibbu11)\ **Post date:** [February 6, 2019, 5:51am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/7 "2019-02-06T05:51:42Z")
I will try to explain the scenario, we are sending at most 10 parallel search queries(by diving the 2 weeks time in 10 buckets). Pasting the sample query we are using : > {"from": 0, "size": 0,"query":{"bool":{"must":[{"range": {"timestamp": > {"gte" : "2019-01-28 00:00","lte":"2019-01:30 00:00","format":"YYYY-MM-dd HH:mm", "time\_zone": "+0530"}}} , > {"bool":{"must": [{"bool":{"should": [{"match": {"group1":{"query":"search\_str", "type": "phrase"}}}, > {"match": {"group2":{"query":"search\_str", "type": "phrase"}}}]}}]}}]}}, > "aggregations": {"Timestamp": > {"terms": {"field": "timestamp","size": 2147483647,"order": { "\_term": "desc"}}, > "aggregations": {"dimension1": {"terms": {"field": "d1","size": 2147483647}, > "aggregations": {"dimension2": {"terms": {"field": "d2","size": 2147483647}, > "aggregations": {"dimension3": {"terms": {"field": "d3","size": 2147483647}, > "aggregations": {"value": {"avg": {"field": "field\_name"}}}}}}}}}}}} Que.1 How does Elasticsearch divide these requests into tasks internally? I didn't find any resources regarding this. Que.2 How can we optimize the query to get similar data in less time? Thinking of using scroll instead of aggregations..which is better? I'm fairly new to ES so pardon me if I have asked any stupid questions. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 6, 2019, 6:58am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/8 "2019-02-06T06:58:08Z")
Are you seeing any evidence in the logs of long or frequent GC? You are specifying very large size parameters for your terms aggregations, which can lead to a lot of unnecessary memory usage. I would recommend tuning this query to try and make it as efficient as possible as you run it frequently. ---
**Author:** ![Shibbu11](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Shibbu11](https://discuss.elastic.co/u/Shibbu11)\ **Post date:** [February 7, 2019, 10:04am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/9 "2019-02-07T10:04:51Z")
Will specifying very large size param affect the memory usage even if we are not getting a large no. of buckets? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 7, 2019, 10:15am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/10 "2019-02-07T10:15:23Z")
Yes, I believe that is still the case. [This blog post](https://www.elastic.co/blog/found-crash-elasticsearch) is quite old, but I believe it is still largely relevant. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2019, 10:15am UTC](https://discuss.elastic.co/t/getting-429-es-exception-while-performing-search-queries/166945/11 "2019-03-07T10:15:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.