# BulkProcessor hangs instead of timeout **URL:** \ **Category:** Elasticsearch\ **Created:** [September 7, 2018, 2:51pm UTC](https://discuss.elastic.co/t/bulkprocessor-hangs-instead-of-timeout/147737 "2018-09-07T14:51:18Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![rk201](https://avatars.discourse-cdn.com/v4/letter/r/4491bb/32.png) [@rk201](https://discuss.elastic.co/u/rk201)\ **Post date:** [September 7, 2018, 2:51pm UTC](https://discuss.elastic.co/t/bulkprocessor-hangs-instead-of-timeout/147737/1 "2018-09-07T14:51:18Z")
Elasticsearch version: 6.1.3, 2 nodes Java Rest High Level Client version: 6.4.0 Bulk size: ~10MB, flush every 5 seconds, concurrentRequests = 1 very similar issues, but without solution: > > > \*\*Elasticsearch version\*\* (\`bin/elasticsearch --version\`): > Tested over all majo…r 5.x versions \[5.1.2 5.2.x 5.3.x ... \] > > \*Plugins installed\*\*: \[defaults \] > > \*\*JVM version\*\* (\`java -version\`): > Java(TM) SE Runtime Environment (build 1.8.0\_144-b01) > > \*\*OS version\*\* (\`uname -a\` if on a Unix-like system): CentOS 6 > > \*\*Description of the problem including expected versus actual behavior\*\*: The issue faced is when using the Java API Transport client in client side applications. If ES Hangs or goes down, all bulk processor threads gets deadlocked. > > This issue was already discussed here: https://discuss.elastic.co/t/java-application-using-bulkprocessing-hangs-if-elasticsearch-hangs/36960/8 > > We are facing this issue as a result of ES going down due to https://github.com/elastic/elasticsearch/issues/24359. > > Can we as a feature implement a timed waiting semaphore as described here: https://discuss.elastic.co/t/java-application-using-bulkprocessing-hangs-if-elasticsearch-hangs/36960/2 and expose it as a param the value for semaphore release. ( Or ofcourse a better way of correcting this ). > > Is there any workaround for this possible from the code outside of driver as a quick fix in case it can't be handled at driver level? > > \*\*Steps to reproduce\*\*: > > 1. ES is running and bulk insertion from application in progress. > 2. ES nodes restart > 3. Deadlock at application side. > > \*\*Thread dump\*\*: > > Looks like as follows: > \`\`\` > "HistoryCachedExecutor-125" #429 daemon prio=5 os\_prio=0 tid=0x00007fd268dea000 nid=0x8fe1 waiting on condition \[0x00007fd0de2e3000\] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for \<0x0000000744a1ca98\> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492) > at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > > Locked ownable synchronizers: > - None > \`\`\` > > Any help in fix or a workaround for this will be really appreciated. Glad to provide any further info needed on this. > [@Java application using BulkProcessing hangs if elasticsearch hangs](https://discuss.elastic.co/t/java-application-using-bulkprocessing-hangs-if-elasticsearch-hangs/36960): > > Using ES 1.7.1 as the server and TransportClient in java application. Java application hangs after some time and becomes unresponsive. We are indexing data using bulk API. In application there are 256 worker threads. Just adding the relevant information from the Thread dump. Out of 256 worker threads, 255 threads are blocked with the following stack dump. "XNIO-1 task-" #325 prio=5 os\_prio=0 tid=0x00007efe6405e800 nid=0x2eea waiting for monitor entry [0x00007effe91d0000] java.lang.Thread.S… My app sometimes hangs when I use BulkProcessor. I checked in JConsole that it happens in BulkRequestHandler.execute(...) when it tries to acquire a semaphore. That semaphore is being released in request's callback - so I think the callback isn't triggered for previous request. I don't see any elastic's messages in log or any exceptions. Simultaneously there is a Logstash instance writing to the same Elastic cluster. It shows up, that when my apps is having above problem, Logstash sometimes logs HTTP 429 errors. But, I thought that BulkProcessor can handle such situation and simply retry the request after waiting some time. In the worst case, I would expect a timeout. What can I do to deal with this? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 5, 2018, 3:04pm UTC](https://discuss.elastic.co/t/bulkprocessor-hangs-instead-of-timeout/147737/2 "2018-10-05T15:04:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.