What does this mean? Do I care? If so what do I do about it?
These happened on what I believe to be a very lightly loaded system in the middle of the night when AFAIK it wasn't doing anything much.
[2017-11-03T00:36:17,884][WARN ][o.e.c.a.s.ShardStateAction] [dev-monitor-2] [er-monitor-2017.11.01][1] received shard failed for shard id [[er-monitor-2017.11.01][1]], allocation id [3lqbKK49RJOd3psmKhg3VQ], primary term [0], message [failed to create shard], failure [IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[er-monitor-2017.11.01][1]: obtaining shard lock timed out after 5000ms]; ]
java.io.IOException: failed to obtain in-memory shard lock
at org.elasticsearch.index.IndexService.createShard(IndexService.java:364) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:499) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:542) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:519) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:204) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.2.jar:5.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.elasticsearch.env.ShardLockObtainFailedException: [er-monitor-2017.11.01][1]: obtaining shard lock timed out after 5000ms
at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:724) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:643) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.index.IndexService.createShard(IndexService.java:294) ~[elasticsearch-5.6.2.jar:5.6.2]
... 17 more
[2017-11-03T00:36:17,935][WARN ][o.e.c.a.s.ShardStateAction] [dev-monitor-2] [er-monitor-2017.11.01][1] received shard failed for shard id [[er-monitor-2017.11.01][1]], allocation id [3lqbKK49RJOd3psmKhg3VQ], primary term [0], message [master {dev-monitor-2}{oRJuzLBKRumqstfiLicChw}{n2U6P-l_QDm_9B9hVArHeg}{172.20.1.136}{172.20.1.136:9300} has not removed previously failed shard. resending shard failure]