# Deadlock in Elastic Search 0.17.4!

**URL:** <https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098>\
**Category:** Elasticsearch\
**Created:** [August 9, 2011, 10:28am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098 "2011-08-09T10:28:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\
**Post date:** [August 9, 2011, 10:28am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/1 "2011-08-09T10:28:08Z")

</div>

Hello. I just upgraded from ES 0.16.2 to 0.17.4.  
After upgrade my tests started to deadlock ALWAYS.  
Below is log from jstack:

# Found one Java-level deadlock:

"elasticsearch[index]-pool-182-thread-1":  
waiting to lock monitor 0x00007ff3cc007150 (object 0x00007ff44e3755b8, a  
java.lang.Object),  
which is held by "elasticsearch[Scarlet  
Beetle]clusterService#updateTask-pool-191-thread-1"  
"elasticsearch[Scarlet Beetle]clusterService#updateTask-pool-191-thread-1":  
waiting to lock monitor 0x00007ff3cc0070a8 (object 0x00007ff450110fd0, a  
java.lang.Object),  
which is held by "elasticsearch[index]-pool-182-thread-1"

# Java stack information for the threads listed above:

"elasticsearch[index]-pool-182-thread-1":  
at  
org.elasticsearch.index.mapper.MapperService$InternalFieldMapperListener.fieldMapper(MapperService.java:710)

- waiting to lock \<0x00007ff44e3755b8\> (a java.lang.Object)  
at  
org.elasticsearch.index.mapper.DocumentMapper.addFieldMapper(DocumentMapper.java:634)
- locked \<0x00007ff450110fd0\> (a java.lang.Object)  
at  
org.elasticsearch.index.mapper.object.ObjectMapper$3.fieldMapper(ObjectMapper.java:695)  
at  
org.elasticsearch.index.mapper.core.AbstractFieldMapper.traverse(AbstractFieldMapper.java:323)  
at  
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:693)
- locked \<0x00007ff450110ae0\> (a java.lang.Object)  
at  
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:440)  
at  
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:566)  
at  
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:289)  
at  
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:185)  
at  
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:428)  
at  
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:341)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:619)  
"elasticsearch[Scarlet Beetle]clusterService#updateTask-pool-191-thread-1":  
at  
org.elasticsearch.index.mapper.DocumentMapper.addObjectMapperListener(DocumentMapper.java:670)
- waiting to lock \<0x00007ff450110fd0\> (a java.lang.Object)  
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:187)
- locked \<0x00007ff44e3755b8\> (a java.lang.Object)  
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)  
at  
org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:382)  
at  
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:349)  
at  
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:176)
- locked \<0x00007ff44dda9ce0\> (a java.lang.Object)  
at  
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:254)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:619)

Found 1 deadlock.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 9, 2011, 10:37am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/2 "2011-08-09T10:37:42Z")

</div>

Can you provide the full thread dump? I only see one thread stack trace  
here. Can you please gist it and not paste it into the mail.

2011/8/9 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)

> Hello. I just upgraded from ES 0.16.2 to 0.17.4.  
> After upgrade my tests started to deadlock ALWAYS.  
> Below is log from jstack:
> 
> # Found one Java-level deadlock:
> 
> "elasticsearch[index]-pool-182-thread-1":  
> waiting to lock monitor 0x00007ff3cc007150 (object 0x00007ff44e3755b8, a  
> java.lang.Object),  
> which is held by "elasticsearch[Scarlet  
> Beetle]clusterService#updateTask-pool-191-thread-1"  
> "elasticsearch[Scarlet Beetle]clusterService#updateTask-pool-191-thread-1":  
> waiting to lock monitor 0x00007ff3cc0070a8 (object 0x00007ff450110fd0, a  
> java.lang.Object),  
> which is held by "elasticsearch[index]-pool-182-thread-1"
> 
> # Java stack information for the threads listed above:
> 
> "elasticsearch[index]-pool-182-thread-1":  
> at  
> org.elasticsearch.index.mapper.MapperService$InternalFieldMapperListener.fieldMapper(MapperService.java:710)
> 
> - waiting to lock \<0x00007ff44e3755b8\> (a java.lang.Object)  
> at  
> org.elasticsearch.index.mapper.DocumentMapper.addFieldMapper(DocumentMapper.java:634)
> - locked \<0x00007ff450110fd0\> (a java.lang.Object)  
> at  
> org.elasticsearch.index.mapper.object.ObjectMapper$3.fieldMapper(ObjectMapper.java:695)  
> at  
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.traverse(AbstractFieldMapper.java:323)  
> at  
> org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:693)
> - locked \<0x00007ff450110ae0\> (a java.lang.Object)  
> at  
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:440)  
> at  
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:566)  
> at  
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)  
> at  
> org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:289)  
> at  
> org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:185)  
> at  
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:428)  
> at  
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:341)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)  
> "elasticsearch[Scarlet Beetle]clusterService#updateTask-pool-191-thread-1":  
> at  
> org.elasticsearch.index.mapper.DocumentMapper.addObjectMapperListener(DocumentMapper.java:670)
> - waiting to lock \<0x00007ff450110fd0\> (a java.lang.Object)  
> at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:187)
> - locked \<0x00007ff44e3755b8\> (a java.lang.Object)  
> at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)  
> at  
> org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:382)  
> at  
> org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:349)  
> at  
> org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:176)
> - locked \<0x00007ff44dda9ce0\> (a java.lang.Object)  
> at  
> org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:254)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)
> 
> Found 1 deadlock.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 9, 2011, 10:40am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/3 "2011-08-09T10:40:43Z")

</div>

Ahh, no need, I see the second thread, lost it in the mail formatting...

On Tue, Aug 9, 2011 at 1:37 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> Can you provide the full thread dump? I only see one thread stack trace  
> here. Can you please gist it and not paste it into the mail.
> 
> 2011/8/9 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)
> 
> > Hello. I just upgraded from ES 0.16.2 to 0.17.4.  
> > After upgrade my tests started to deadlock ALWAYS.  
> > Below is log from jstack:
> > 
> > # Found one Java-level deadlock:
> > 
> > "elasticsearch[index]-pool-182-thread-1":  
> > waiting to lock monitor 0x00007ff3cc007150 (object 0x00007ff44e3755b8, a  
> > java.lang.Object),  
> > which is held by "elasticsearch[Scarlet  
> > Beetle]clusterService#updateTask-pool-191-thread-1"  
> > "elasticsearch[Scarlet  
> > Beetle]clusterService#updateTask-pool-191-thread-1":  
> > waiting to lock monitor 0x00007ff3cc0070a8 (object 0x00007ff450110fd0, a  
> > java.lang.Object),  
> > which is held by "elasticsearch[index]-pool-182-thread-1"
> > 
> > # Java stack information for the threads listed above:
> > 
> > "elasticsearch[index]-pool-182-thread-1":  
> > at  
> > org.elasticsearch.index.mapper.MapperService$InternalFieldMapperListener.fieldMapper(MapperService.java:710)
> > 
> > - waiting to lock \<0x00007ff44e3755b8\> (a java.lang.Object)  
> > at  
> > org.elasticsearch.index.mapper.DocumentMapper.addFieldMapper(DocumentMapper.java:634)
> > - locked \<0x00007ff450110fd0\> (a java.lang.Object)  
> > at  
> > org.elasticsearch.index.mapper.object.ObjectMapper$3.fieldMapper(ObjectMapper.java:695)  
> > at  
> > org.elasticsearch.index.mapper.core.AbstractFieldMapper.traverse(AbstractFieldMapper.java:323)  
> > at  
> > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:693)
> > - locked \<0x00007ff450110ae0\> (a java.lang.Object)  
> > at  
> > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:440)  
> > at  
> > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:566)  
> > at  
> > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)  
> > at  
> > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:289)  
> > at  
> > org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:185)  
> > at  
> > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:428)  
> > at  
> > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:341)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:619)  
> > "elasticsearch[Scarlet  
> > Beetle]clusterService#updateTask-pool-191-thread-1":  
> > at  
> > org.elasticsearch.index.mapper.DocumentMapper.addObjectMapperListener(DocumentMapper.java:670)
> > - waiting to lock \<0x00007ff450110fd0\> (a java.lang.Object)  
> > at  
> > org.elasticsearch.index.mapper.MapperService.add(MapperService.java:187)
> > - locked \<0x00007ff44e3755b8\> (a java.lang.Object)  
> > at  
> > org.elasticsearch.index.mapper.MapperService.add(MapperService.java:166)  
> > at  
> > org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:382)  
> > at  
> > org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:349)  
> > at  
> > org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:176)
> > - locked \<0x00007ff44dda9ce0\> (a java.lang.Object)  
> > at  
> > org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:254)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:619)
> > 
> > Found 1 deadlock.

---

<div class="post-metadata">

**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\
**Post date:** [August 9, 2011, 10:44am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/4 "2011-08-09T10:44:48Z")

</div>

If You need stack traces of all threads, here it is:

> <https://gist.github.com/Avatah/1133740>

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 9, 2011, 11:27am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/5 "2011-08-09T11:27:41Z")

</div>

Can you open an issue for this while I work on a fix. Its been there in  
previous versions, its just a matter of timing and order. Small chances for  
it to happen, but it needs to be fixed.

2011/8/9 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)

> If You need stack traces of all threads, here it is:  
> [Deadlock in ES 0.17.4 · GitHub](https://gist.github.com/1133740)

---

<div class="post-metadata">

**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\
**Post date:** [August 9, 2011, 12:26pm UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/6 "2011-08-09T12:26:06Z")

</div>

Issue created: [https://github.com/elasticsearch/elasticsearch/issues/1221](https://github.com/elasticsearch/elasticsearch/issues/1221)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 9, 2011, 12:27pm UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/7 "2011-08-09T12:27:33Z")

</div>

Argh :), already created an issue:  
[Rare deadlock when introducing new mapping fields/objects · Issue #1222 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1222) since I have the  
fix ready, will point your issue to the other one...

2011/8/9 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)

> Issue created: [Deadlock in Elastic Search 0.17.4 · Issue #1221 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1221)

---

<div class="post-metadata">

**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\
**Post date:** [August 9, 2011, 12:55pm UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/8 "2011-08-09T12:55:26Z")

</div>

Thank You.  
Do you plan to release version 0.17.5 with this fix? When?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 9, 2011, 2:26pm UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/9 "2011-08-09T14:26:48Z")

</div>

Yes, it has been applied to 0.17 branch and will be part of 0.17.5. It will  
be released in a few days.

2011/8/9 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)

> Thank You.  
> Do you plan to release version 0.17.5 with this fix? When?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:57am UTC](https://discuss.elastic.co/t/deadlock-in-elastic-search-0-17-4/5098/10 "2017-07-06T03:57:46Z")

</div>