# Blocked Thread Problem **URL:** \ **Category:** Elasticsearch\ **Created:** [December 13, 2010, 1:55pm UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656 "2010-12-13T13:55:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\ **Post date:** [December 13, 2010, 1:55pm UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/1 "2010-12-13T13:55:49Z")
Hi, I am using elasticsearch 0.13.0. I use TransportClient. When I exceuted get action agains ES one of my threads are blocked. I got the following thread dump - parking to wait for \<0x22b00c80\> (a org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239) at org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:227) at org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:68) at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:59) When I executed same action over HTTP, it works as expected. Do you have any idea about this problem? -- Mustafa Sener [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 13, 2010, 6:17pm UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/2 "2010-12-13T18:17:29Z")
Hi, There is no timeout on the get operation, it will either get a response, or, when the socket connection breaks, it will bail with an exception. This might happen because of several reasons, is there a chance that this can be recreated (I know its probably really hard...)... ? -shay.banon On Monday, December 13, 2010 at 3:55 PM, Mustafa Sener wrote: > Hi, > I am using elasticsearch 0.13.0. I use TransportClient. When I exceuted get action agains ES one of my threads are blocked. I got the following thread dump > > - parking to wait for \<0x22b00c80\> (a org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947) > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239) > at org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:227) > at org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:68) > at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) > at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:59) > > When I executed same action over HTTP, it works as expected. Do you have any idea about this problem? > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) > > Attachments: > > - threaddump.txt ---
**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\ **Post date:** [December 13, 2010, 8:22pm UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/3 "2010-12-13T20:22:20Z")
Actually I can recreate this problem almost every time when I run the automated tests of our product. But there is a huge amount of code around ES integration with our product. I will try to simulate this problem just by using TransportClient and ES server. On Mon, Dec 13, 2010 at 8:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hi, > > There is no timeout on the get operation, it will either get a response, > or, when the socket connection breaks, it will bail with an exception. This > might happen because of several reasons, is there a chance that this can be > recreated (I know its probably really hard...)... ? > > -shay.banon > > On Monday, December 13, 2010 at 3:55 PM, Mustafa Sener wrote: > > Hi, > I am using elasticsearch 0.13.0. I use TransportClient. When I exceuted > get action agains ES one of my threads are blocked. I got the following > thread dump > > - parking to wait for \<0x22b00c80\> (a > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239) > at > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:227) > at > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:68) > at > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) > at > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:59) > > When I executed same action over HTTP, it works as expected. Do you have > any idea about this problem? > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) > > Attachments: > > - threaddump.txt -- Mustafa Sener [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 13, 2010, 8:24pm UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/4 "2010-12-13T20:24:06Z")
Cool, thats great (the ability to recreate it)!. Thanks for the effort in trying to simplify this!. On Monday, December 13, 2010 at 10:22 PM, Mustafa Sener wrote: > Actually I can recreate this problem almost every time when I run the automated tests of our product. But there is a huge amount of code around ES integration with our product. I will try to simulate this problem just by using TransportClient and ES server. > > On Mon, Dec 13, 2010 at 8:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Hi, > > > > There is no timeout on the get operation, it will either get a response, or, when the socket connection breaks, it will bail with an exception. This might happen because of several reasons, is there a chance that this can be recreated (I know its probably really hard...)... ? > > > > -shay.banon > > > > On Monday, December 13, 2010 at 3:55 PM, Mustafa Sener wrote: > > > > > Hi, > > > I am using elasticsearch 0.13.0. I use TransportClient. When I exceuted get action agains ES one of my threads are blocked. I got the following thread dump > > > > > > - parking to wait for \<0x22b00c80\> (a org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947) > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239) > > > at org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:227) > > > at org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:68) > > > at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) > > > at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:59) > > > > > > When I executed same action over HTTP, it works as expected. Do you have any idea about this problem? > > > > > > -- > > > Mustafa Sener > > > [www.ifountain.com](http://www.ifountain.com) > > > > > > Attachments: > > > > > > - threaddump.txt > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![Mustafa\_Sener](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@Mustafa\_Sener](https://discuss.elastic.co/u/Mustafa_Sener)\ **Post date:** [December 14, 2010, 9:36am UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/5 "2010-12-14T09:36:41Z")
Hi Shay, I figured out that this problem occurs as a result of our usage of async mechanism. If we have an ActionListener waits on a lock and also we have a sync request without a timeout and accessing same lock, a deadlock occurs. I tried to simulate the situation below =============================================== Object lock = new Object(); client.get(new GetRequest(....), new ActionListener{ synchronized(lock ){ ............... } }) # synchronized(lock ){ client.get(new GetRequest(....)).actionGet(); } Regards.. On Mon, Dec 13, 2010 at 10:24 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Cool, thats great (the ability to recreate it)!. Thanks for the effort in > trying to simplify this!. > > On Monday, December 13, 2010 at 10:22 PM, Mustafa Sener wrote: > > Actually I can recreate this problem almost every time when I run the > automated tests of our product. But there is a huge amount of code around ES > integration with our product. I will try to simulate this problem just by > using TransportClient and ES server. > > On Mon, Dec 13, 2010 at 8:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Hi, > > There is no timeout on the get operation, it will either get a response, > or, when the socket connection breaks, it will bail with an exception. This > might happen because of several reasons, is there a chance that this can be > recreated (I know its probably really hard...)... ? > > -shay.banon > > On Monday, December 13, 2010 at 3:55 PM, Mustafa Sener wrote: > > Hi, > I am using elasticsearch 0.13.0. I use TransportClient. When I exceuted > get action agains ES one of my threads are blocked. I got the following > thread dump > > - parking to wait for \<0x22b00c80\> (a > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239) > at > org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:227) > at > org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:68) > at > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) > at > org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:59) > > When I executed same action over HTTP, it works as expected. Do you have > any idea about this problem? > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) > > Attachments: > > - threaddump.txt > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) -- Mustafa Sener [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 14, 2010, 4:54pm UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/6 "2010-12-14T16:54:51Z")
cool, probably was a nasty one to find... On Tuesday, December 14, 2010 at 11:36 AM, Mustafa Sener wrote: > Hi Shay, > I figured out that this problem occurs as a result of our usage of async mechanism. If we have an ActionListener waits on a lock and also we have a sync request without a timeout and accessing same lock, a deadlock occurs. I tried to simulate the situation below > > =============================================== > Object lock = new Object(); > client.get(new GetRequest(....), new ActionListener{ > synchronized(lock ){ > ............... > } > }) > > # synchronized(lock ){ client.get(new GetRequest(....)).actionGet(); } > > Regards.. > > On Mon, Dec 13, 2010 at 10:24 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Cool, thats great (the ability to recreate it)!. Thanks for the effort in trying to simplify this!. > > > > On Monday, December 13, 2010 at 10:22 PM, Mustafa Sener wrote: > > > > > Actually I can recreate this problem almost every time when I run the automated tests of our product. But there is a huge amount of code around ES integration with our product. I will try to simulate this problem just by using TransportClient and ES server. > > > > > > On Mon, Dec 13, 2010 at 8:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Hi, > > > > > > > > There is no timeout on the get operation, it will either get a response, or, when the socket connection breaks, it will bail with an exception. This might happen because of several reasons, is there a chance that this can be recreated (I know its probably really hard...)... ? > > > > > > > > -shay.banon > > > > > > > > On Monday, December 13, 2010 at 3:55 PM, Mustafa Sener wrote: > > > > > > > > > Hi, > > > > > I am using elasticsearch 0.13.0. I use TransportClient. When I exceuted get action agains ES one of my threads are blocked. I got the following thread dump > > > > > > > > > > - parking to wait for \<0x22b00c80\> (a org.elasticsearch.common.util.concurrent.AbstractFuture$Sync) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:947) > > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1239) > > > > > at org.elasticsearch.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:227) > > > > > at org.elasticsearch.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:68) > > > > > at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:68) > > > > > at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:59) > > > > > > > > > > When I executed same action over HTTP, it works as expected. Do you have any idea about this problem? > > > > > > > > > > -- > > > > > Mustafa Sener > > > > > [www.ifountain.com](http://www.ifountain.com) > > > > > > > > > > Attachments: > > > > > > > > > > - threaddump.txt > > > > > > -- > > > Mustafa Sener > > > [www.ifountain.com](http://www.ifountain.com) > > -- > Mustafa Sener > [www.ifountain.com](http://www.ifountain.com) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/blocked-thread-problem/3656/7 "2017-07-06T04:15:20Z")