# How to diagnose when indexing performance suddenly droped **URL:** \ **Category:** Elasticsearch\ **Created:** [May 13, 2020, 12:40pm UTC](https://discuss.elastic.co/t/how-to-diagnose-when-indexing-performance-suddenly-droped/232436 "2020-05-13T12:40:26Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![111336](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111336/32/68208_2.png) [@111336](https://discuss.elastic.co/u/111336)\ **Post date:** [May 13, 2020, 12:40pm UTC](https://discuss.elastic.co/t/how-to-diagnose-when-indexing-performance-suddenly-droped/232436/1 "2020-05-13T12:40:26Z")
Hi. I'm running a es cluster which has 50 data nodes, 20 for a logging index, 20 for a data index, 10 for other indices, and 4 coordinators and 3 master nodes. One day the indexing performance of the logging index was suddenly dropped. - es 7.4.0 + docker environment - cpus was used about 10% - gc was less than 5ms - ssd disk io was about 20MB/s - hot threads didn't exists in most logging data nodes. -- [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) - tasks manager show lot of actions 'indices:data/write/bulk[s]' or '[s][p]' in both coordinator and data nodes with high running\_time\_in\_nanos higher than 5,000,000,000 -- all coordinator nodes : lot of status were rerouted -- data nodes of logging index : log of status were waiting\_on\_primary, primary. shard numbers were different -- other data nodes : few indexing action with low running\_time\_in\_nanos After I restart containers, it back to normal. - cpus went high closed to 80% - gc was about 20ms - ssd disk io was abount 450MB/s - no task with long running\_time\_in\_nanos It looked like indexing was stuck somewhere between requests of indexing accepted and writing it into shard. I don't know how to figure it out. Is there any tool to use? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 13, 2020, 11:39pm UTC](https://discuss.elastic.co/t/how-to-diagnose-when-indexing-performance-suddenly-droped/232436/2 "2020-05-13T23:39:54Z")
Welcome 🙂 > [@111336](#): > > I'm running a es cluster which has 50 data nodes, 20 for a logging index, 20 for a data index, 10 for other indices Ideally you should split those use cases out, so that they don't impact each other and then the users of each dataset. Do you have [Monitoring](https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-overview.html) enabled? ---
**Author:** ![111336](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111336/32/68208_2.png) [@111336](https://discuss.elastic.co/u/111336)\ **Post date:** [May 14, 2020, 2:27am UTC](https://discuss.elastic.co/t/how-to-diagnose-when-indexing-performance-suddenly-droped/232436/3 "2020-05-14T02:27:14Z")
Thank to reply. While indexing of the logging data got slow, other indexing of indices hadn't any problem. It would be good to split cluster for each purpose, but allocating index sperately was enough. All metric are collected using fluentd. I looked into machine cpu, memory, io of all es nodes. Since the task manager api said indexing was slow, I guess logstash was not a candidate causes of problems. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 11, 2020, 2:27am UTC](https://discuss.elastic.co/t/how-to-diagnose-when-indexing-performance-suddenly-droped/232436/4 "2020-06-11T02:27:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.