# How to debug Timeout Error **URL:** \ **Category:** Elasticsearch\ **Created:** [March 19, 2019, 11:22am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950 "2019-03-19T11:22:21Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![harimenonm](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@harimenonm](https://discuss.elastic.co/u/harimenonm)\ **Post date:** [March 19, 2019, 11:22am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/1 "2019-03-19T11:22:22Z")
Hi All, I am getting the following error when performing a bulk insert.Caused by: java.io.IOException: listener timeout after waiting for [30000] ms at elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:699) at elasticsearch.client.RestClient.performRequest(RestClient.java:224) at elasticsearch.client.RestClient.performRequest(RestClient.java:196) What are the steps to debug this issue ? In the Elastic search logs I am not seeing any error. Is there anyway we can identify the reason behind the timeout? Should we enable some specific properties to enable the logs? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 19, 2019, 11:28am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/2 "2019-03-19T11:28:14Z")
What is the specification of your cluster? Is it under heavy load? What is the size of your requests? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 19, 2019, 11:35am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/3 "2019-03-19T11:35:55Z")
What is this package `com.oracle.es.elasticsearch.client.RestClient`? ---
**Author:** ![harimenonm](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@harimenonm](https://discuss.elastic.co/u/harimenonm)\ **Post date:** [March 21, 2019, 9:07am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/4 "2019-03-21T09:07:30Z")
We have a two node setup with 12 GB each. It is under heavy load. We are pushing 100 documents at a time each doc having 20kb in average. See the elastic logs - [https://drive.google.com/file/d/1vZbjfByxZ0oiZ11a5evdcahDP4MttbCA/view?usp=sharing](https://drive.google.com/file/d/1vZbjfByxZ0oiZ11a5evdcahDP4MttbCA/view?usp=sharing) ---
**Author:** ![harimenonm](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@harimenonm](https://discuss.elastic.co/u/harimenonm)\ **Post date:** [March 21, 2019, 9:08am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/5 "2019-03-21T09:08:04Z")
Its our code from where i am sending the bulk request. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 21, 2019, 9:12am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/6 "2019-03-21T09:12:08Z")
What appears to be limiting performance? Is CPU maxed out? Are you seeing a lot of iowait due to potentially slow storage? Any indications in the logs of slow merging or long and/or frequent GC? ---
**Author:** ![harimenonm](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@harimenonm](https://discuss.elastic.co/u/harimenonm)\ **Post date:** [March 21, 2019, 10:19am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/7 "2019-03-21T10:19:48Z")
When we enable slow logs we can see some documents taking time. I checked the heap, and only 60% is used. How can we check f its because of slow merging or iowait? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 21, 2019, 10:43am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/8 "2019-03-21T10:43:27Z")
Use `iostat` on the data nodes. ---
**Author:** ![manucet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manucet/32/42594_2.png) [@manucet](https://discuss.elastic.co/u/manucet)\ **Post date:** [March 22, 2019, 2:52pm UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/9 "2019-03-22T14:52:03Z")
> [@Christian\_Dahlqvist](#): > > tentially slow storage? Any indications in the logs of slow merging or long and/or frequent GC? I am seeing a similar issue and can see the following in the elastic server logs when the timeouts occur [2019-03-22T14:40:08,361][DEBUG][o.e.i.e.InternalEngine$EngineMergeScheduler] [ElasticServer1] [mm\_110fe1d3-13cb-4d3c-aec8-771cc04a789c\_d53dc9e5-3132-4b31-bd9a-24609f1b2334][2] merge segment [\_w] done: took [2m], [627.1 MB], [624,779 docs], [0s stopped], [13.1s throttled], [613.9 MB written], [18.2 MB/sec throttle] [2019-03-22T14:40:40,437][DEBUG][o.e.i.e.InternalEngine$EngineMergeScheduler] [ElasticServer1] [mm\_110fe1d3-13cb-4d3c-aec8-771cc04a789c\_d53dc9e5-3132-4b31-bd9a-24609f1b2334][4] merge segment [\_v] done: took [2.4m], [793.0 MB], [776,320 docs], [0s stopped], [17.7s throttled], [781.3 MB written], [18.2 MB/sec throttle] [2019-03-22T14:40:41,615][DEBUG][o.e.m.j.JvmGcMonitorService] [ElasticServer1] [gc][245344] overhead, spent [121ms] collecting in the last [1s] [2019-03-22T14:40:43,638][DEBUG][o.e.m.j.JvmGcMonitorService] [ElasticServer1] [gc][245346] overhead, spent [108ms] collecting in the last [1s] [2019-03-22T14:40:58,664][DEBUG][o.e.m.j.JvmGcMonitorService] [ElasticServer1] [gc][245361] overhead, spent [103ms] collecting in the last [1s] [2019-03-22T14:41:01,018][DEBUG][o.e.i.e.InternalEngine$EngineMergeScheduler] [ElasticServer1] [mm\_110fe1d3-13cb-4d3c-aec8-771cc04a789c\_d53dc9e5-3132-4b31-bd9a-24609f1b2334][0] merge segment [\_13] done: took [1.6m], [533.6 MB], [511,864 docs], [0s stopped], [13.2s throttled], [530.5 MB written], [16.5 MB/sec throttle] When this occurs will the bulk indexing get slowed down resulting in timeouts? It looks so from the slowlogs collected in a previous run. I did try out invoking iostat during the process but did not see much iowait, however the segment merge happened after I invoked iostat so its possible there was a slowdown during the merge. What would be your recommendation to prevent these timeouts? Should i be increasing the timeout or reducing the number of threads that are currently pushing data during indexing or both? Also assume that I use a single thread to push data, even then at some point of time there will be a segment merge happening and if it again takes 1 to 2 minutes as above and slows down the indexing failure can still occur. So what is the recommended way out of this? I want indexing to not fail and some reduction in indexing speed is not a problem. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2019, 6:51am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/10 "2019-03-23T06:51:18Z")
It looks like segment merging can't keep up, which is a sign you likely have very slow storage that is the bottleneck. You should have a look at [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/tune-for-indexing-speed.html). In older versions there used to be parameters related to the number of merging threads that needed to be tuned, but that has since been automated. The best way to get rid of this problem is however to upgrade to faster and more performant storage. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 20, 2019, 6:51am UTC](https://discuss.elastic.co/t/how-to-debug-timeout-error/172950/11 "2019-04-20T06:51:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.