# Failed to create shard, failure IOException\[failed to obtain in-memory shard lock\]
**URL:** https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323
**Category:** Elasticsearch
**Tags:** docker
**Created:** [July 23, 2020, 10:27am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323 "2020-07-23T10:27:53Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@AaronSarkissian](https://discuss.elastic.co/u/AaronSarkissian)
#### Post date: [July 23, 2020, 10:27am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/1 "2020-07-23T10:27:53Z")
My cluster setup is 2 nodes, both docker based on different VMs in the same network.
My cluster health becomes yellow after few hours it start to get unassigned status one by one, till after a day all the replica shards become unassigned and when I check the shard allocation it looks like this:
So call the following command: `POST /_cluster/reroute?retry_failed=true`
Immediately the shards are starting to initialize:
After like 3-4 minutes, it looks like all assigned, and the cluster health is green:
So, I started using the allocation/explain API: `GET /_cluster/allocation/explain?pretty`
And I got:
```auto
{
"index" : "projects",
"shard" : 4,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "MANUAL_ALLOCATION",
"at" : "2020-07-21T08:22:48.307Z",
"details" : "failed shard on node [Vnl1IdQOTdGDZcr0qG1Wxw]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[projects][4]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "awaiting_info",
"allocate_explanation" : "cannot allocate because information about existing shard data is still being retrieved from some of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "Vnl1IdQOTdGDZcr0qG1Wxw",
"node_name" : "eu01",
"transport_address" : "172.18.4.6:9300",
"node_decision" : "throttled",
"deciders" : [
{
"decider" : "throttling",
"decision" : "THROTTLE",
"explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node_concurrent_incoming_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
}
]
},
{
"node_id" : "gI3ylY0JTNWuCSOSJ1vN2g",
"node_name" : "us01",
"transport_address" : "172.18.1.11:9300",
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "a copy of this shard is already allocated to this node [[projects][4], node[gI3ylY0JTNWuCSOSJ1vN2g], [P], s[STARTED], a[id=X-D0rlNmRmuTSkWlR3AQ7w]]"
},
{
"decider" : "throttling",
"decision" : "THROTTLE",
"explanation" : "reached the limit of outgoing shard recoveries [2] on the node [gI3ylY0JTNWuCSOSJ1vN2g] which holds the primary, cluster setting [cluster.routing.allocation.node_concurrent_outgoing_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
}
]
}
]
}
```
I checked my disk space, it's 90% free, so that is the case here.
Can someone help me understand what is the issue here and why the shards are getting unassigned every day?
Thanks
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 23, 2020, 10:29am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/2 "2020-07-23T10:29:44Z")
What type of storage are you using?
---
### Author:  [@AaronSarkissian](https://discuss.elastic.co/u/AaronSarkissian)
#### Post date: [July 23, 2020, 10:31am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/3 "2020-07-23T10:31:19Z")
Premium SSDs
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 23, 2020, 10:57am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/4 "2020-07-23T10:57:13Z")
Where is this hosted? Could there be connectivity issues?
---
### Author:  [@AaronSarkissian](https://discuss.elastic.co/u/AaronSarkissian)
#### Post date: [July 23, 2020, 10:59am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/5 "2020-07-23T10:59:48Z")
Azure memory optimized VMs. The network should be stable I think.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 23, 2020, 11:05am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/6 "2020-07-23T11:05:27Z")
Your nodes are called `us01` and `eu01`. Are they respectively in the US and the EU?
---
### Author:  [@AaronSarkissian](https://discuss.elastic.co/u/AaronSarkissian)
#### Post date: [July 23, 2020, 11:07am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/7 "2020-07-23T11:07:35Z")
Yes, exactly. The ping between them is around 80ms
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 23, 2020, 11:14am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/8 "2020-07-23T11:14:32Z")
Ok seems like the expected behaviour then, transatlantic networking isn't nearly fast or reliable enough for this.
---
### Author:  [@AaronSarkissian](https://discuss.elastic.co/u/AaronSarkissian)
#### Post date: [July 23, 2020, 11:18am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/9 "2020-07-23T11:18:38Z")
Hmm. Can it be because of the following settings in the jvm.options config?
```auto
## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10
```
And the full file is here:
```auto
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms8g
-Xmx8g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
# 10-:-XX:+UseG1GC
# 10-:-XX:InitiatingHeapOccupancyPercent=75
## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# explicitly set the stack size
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 23, 2020, 11:26am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/10 "2020-07-23T11:26:49Z")
No, I don't see how adjusting the DNS caching config (or indeed any other settings) can change the fact that transatlantic networking isn't nearly fast or reliable enough for this. Clusters should be contained in a single datacenter, maybe with [remote clusters](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/modules-remote-clusters.html) elsewhere in the world.
---
### Author:  [@AaronSarkissian](https://discuss.elastic.co/u/AaronSarkissian)
#### Post date: [July 23, 2020, 11:31am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/11 "2020-07-23T11:31:58Z")
Ok, I'll try that see if it helps. But meanwhile I wonder why it happens only to the index with more than one shards. Because the `devicelocations` index never gets an unassigned status.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 20, 2020, 11:32am UTC](https://discuss.elastic.co/t/failed-to-create-shard-failure-ioexception-failed-to-obtain-in-memory-shard-lock/242323/12 "2020-08-20T11:32:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.