# ElasticSearch Read/Write performance issue on Peak Load **URL:** \ **Category:** Elasticsearch\ **Created:** [January 24, 2022, 10:48am UTC](https://discuss.elastic.co/t/elasticsearch-read-write-performance-issue-on-peak-load/295216 "2022-01-24T10:48:34Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sahil441](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sahil441/32/100185_2.png) [@sahil441](https://discuss.elastic.co/u/sahil441)\ **Post date:** [January 24, 2022, 10:48am UTC](https://discuss.elastic.co/t/elasticsearch-read-write-performance-issue-on-peak-load/295216/1 "2022-01-24T10:48:34Z")
**Context:** 1. We are running ES Cluster (ES Version - "7.9.1") with 3 Master-Nodes & 8 data-nodes (AWS-"r5.2xlarge") 2. We are using the default ES thread pool configuration 3. There are 2 indices having 5 shards each (used by the given microservice) in which our data is stored, having 5 shards each 4. Another Microservices are utilizing the same ES Cluster (having their separate indices) **Issue:** At peak load, many read & write requests on ES are timed-out (client-side socket timeout is very high 12s) **Cause of Issue:** At peak load, lots of read/write requests are getting queued up waiting for a worker thread from the corresponding thread pool, hence the latency **Solutions:** 1. Increase to number of shards to at least the number of data nodes 2. Have a separate number Cluster for this service so that resources are not shared by other services **Questions:** 1. Can we increase the number of search & write threads in the pool by reducing the size of some other thread pools for e.g 'sql-write (we don't use it) & reducing the core size of many dynamic pools to 0 (which we don't generally use) ? 2. Any other recommendations? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 24, 2022, 11:16am UTC](https://discuss.elastic.co/t/elasticsearch-read-write-performance-issue-on-peak-load/295216/2 "2022-01-24T11:16:51Z")
First try to identify what is the bottleneck. Indexing is quite I/O intensive, so I would first look at storage performance and iowait. What type of storage are you using? What is the load on it? How many indices and shards are you actively indexing into (in the cluster as a whole). ---
**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\ **Post date:** [January 24, 2022, 12:42pm UTC](https://discuss.elastic.co/t/elasticsearch-read-write-performance-issue-on-peak-load/295216/3 "2022-01-24T12:42:45Z")
What is the load (qps and wps) at which you start seeing this performance degradation? ---
**Author:** ![sahil441](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sahil441/32/100185_2.png) [@sahil441](https://discuss.elastic.co/u/sahil441)\ **Post date:** [January 26, 2022, 3:26pm UTC](https://discuss.elastic.co/t/elasticsearch-read-write-performance-issue-on-peak-load/295216/5 "2022-01-26T15:26:21Z")
Storage Type is "Elastic Block Storage" And "EBS volume type" is SSD (general Purpose) **Read Latency:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/8/18240faa10d8f3f8865c593571a1b7ba130f8a93.png) **Write Latency:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/7/4702d56e45dc328f00dc4993d739fded9a5385e2.png) **Read Throuput:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6cd6a806e6a29921cb5e78159b3c22956cf18640.png) **Write througput:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9c1ea30954ef2c1fabf1d80a329d389f8f458f73.png) **Disk Queue depth:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a332be2bde3a82728afb7affb59b7154c26d7933.png) **Read IOPS:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9c184d1f6b5350b465382dff556c881e5fec9a9.png) **Write IOPS:** ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/eee114999de134acf2816362dfca4f65815c2753.png) Past 2 weeks stats ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 23, 2022, 3:26pm UTC](https://discuss.elastic.co/t/elasticsearch-read-write-performance-issue-on-peak-load/295216/6 "2022-02-23T15:26:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.