# Help needed for tracing index slow down **URL:** \ **Category:** Elasticsearch\ **Created:** [November 13, 2012, 6:56pm UTC](https://discuss.elastic.co/t/help-needed-for-tracing-index-slow-down/9701 "2012-11-13T18:56:19Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Jae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jae/32/791_2.png) [@Jae](https://discuss.elastic.co/u/Jae)\ **Post date:** [November 13, 2012, 6:56pm UTC](https://discuss.elastic.co/t/help-needed-for-tracing-index-slow-down/9701/1 "2012-11-13T18:56:19Z")
I am using elasticsearch 0.19.11 at AWS EC2 m2.4xlarge 6 instances, allocated 35GB to ES with mlock.all=true Now, I implemented kafka river to pull the data from kafka cluster. There can be index delay between current timestamp and the timestamp of the pulled data depending on elasticsearch index performance. This measure is very critical because we are using elasticsearch as realtime log analytics. I can see index delay is increasing significantly at some point but I don't know how to trace the problem. [https://lh5.googleusercontent.com/-fAJKLSDRL9k/UKKXCcrZ2RI/AAAAAAAAADo/Gx1Iow31ECw/s1600/indexdelay.tiff](https://lh5.googleusercontent.com/-fAJKLSDRL9k/UKKXCcrZ2RI/AAAAAAAAADo/Gx1Iow31ECw/s1600/indexdelay.tiff) I figure this is caused by merging but I am suspicious about disk IO behavior. I set up elasticsearch data directory at disk md0 but index delay looks synched with sda1 mounted at root(/) not md0. [https://lh5.googleusercontent.com/-apgN-Uvexi4/UKKXZ6WXY1I/AAAAAAAAADw/5iFJjOaecpo/s1600/disk\_md0.tiff](https://lh5.googleusercontent.com/-apgN-Uvexi4/UKKXZ6WXY1I/AAAAAAAAADw/5iFJjOaecpo/s1600/disk_md0.tiff) [https://lh6.googleusercontent.com/-NSGir6ksLLQ/UKKXcdTSc1I/AAAAAAAAAD4/431bqo-uHhM/s1600/disk\_sda1.tiff](https://lh6.googleusercontent.com/-NSGir6ksLLQ/UKKXcdTSc1I/AAAAAAAAAD4/431bqo-uHhM/s1600/disk_sda1.tiff) Can you guess what is writing to sda1? Also, how can I trace this slow down? I want to see any indexing actions such as when merging, flushing, refresh starts but I cannot see any INFO messages with the following logging.yml setting. ootLogger: INFO, file logger: # log action execution errors for easier debugging # gateway index.gateway: INFO # peer shard recovery indices.recovery: INFO # discovery discovery: INFO index.search.slowlog: INFO, index\_search\_slow\_log\_file additivity: index.search.slowlog: false Thank you! Best, Jae -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [November 14, 2012, 10:51am UTC](https://discuss.elastic.co/t/help-needed-for-tracing-index-slow-down/9701/2 "2012-11-14T10:51:46Z")
Hello Jae, I would start with some monitoring tool for ES, to see if it brings up any clues. There are quite a lot out there, we also provide a good one that's still free[0] If you want specific stats about merging, refreshing and flushing, check out the Indices Stats API[1] And if you want to know what is writing to /dev/sda1 (I can't think of anything else but the log file), a good starting point is the fuser command. Something like: # fuser -m /dev/sda1 [0] [Elasticsearch - Sematext Documentation](http://sematext.com/spm/elasticsearch-performance-monitoring/index.html) [1] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-stats.html) ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Tue, Nov 13, 2012 at 8:56 PM, Jae [metacret@gmail.com](mailto:metacret@gmail.com) wrote: > I am using elasticsearch 0.19.11 at AWS EC2 m2.4xlarge 6 instances, > allocated 35GB to ES with mlock.all=true > > Now, I implemented kafka river to pull the data from kafka cluster. There > can be index delay between current timestamp and the timestamp of the > pulled data depending on elasticsearch index performance. This measure is > very critical because we are using elasticsearch as realtime log analytics. > I can see index delay is increasing significantly at some point but I don't > know how to trace the problem. > > [https://lh5.googleusercontent.com/-fAJKLSDRL9k/UKKXCcrZ2RI/AAAAAAAAADo/Gx1Iow31ECw/s1600/indexdelay.tiff](https://lh5.googleusercontent.com/-fAJKLSDRL9k/UKKXCcrZ2RI/AAAAAAAAADo/Gx1Iow31ECw/s1600/indexdelay.tiff) > > I figure this is caused by merging but I am suspicious about disk IO > behavior. I set up elasticsearch data directory at disk md0 but index delay > looks synched with sda1 mounted at root(/) not md0. > > [https://lh5.googleusercontent.com/-apgN-Uvexi4/UKKXZ6WXY1I/AAAAAAAAADw/5iFJjOaecpo/s1600/disk\_md0.tiff](https://lh5.googleusercontent.com/-apgN-Uvexi4/UKKXZ6WXY1I/AAAAAAAAADw/5iFJjOaecpo/s1600/disk_md0.tiff) > > [https://lh6.googleusercontent.com/-NSGir6ksLLQ/UKKXcdTSc1I/AAAAAAAAAD4/431bqo-uHhM/s1600/disk\_sda1.tiff](https://lh6.googleusercontent.com/-NSGir6ksLLQ/UKKXcdTSc1I/AAAAAAAAAD4/431bqo-uHhM/s1600/disk_sda1.tiff) > > Can you guess what is writing to sda1? Also, how can I trace this slow > down? I want to see any indexing actions such as when merging, flushing, > refresh starts but I cannot see any INFO messages with the following > logging.yml setting. > > ootLogger: INFO, file > logger: > > # log action execution errors for easier debugging > > # gateway > > index.gateway: INFO > > # peer shard recovery > > indices.recovery: INFO > > # discovery > > discovery: INFO > > index.search.slowlog: INFO, index\_search\_slow\_log\_file > > additivity: > index.search.slowlog: false > > Thank you! > Best, Jae > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/help-needed-for-tracing-index-slow-down/9701/3 "2017-07-06T03:04:25Z")
Closed on July 6, 2017, 3:04am UTC