# Shard lock issue **URL:** \ **Category:** Elasticsearch\ **Created:** [January 27, 2023, 8:29am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094 "2023-01-27T08:29:10Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![akihu](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@akihu](https://discuss.elastic.co/u/akihu)\ **Post date:** [January 27, 2023, 8:29am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/1 "2023-01-27T08:29:11Z")
Dear Community, Would you happen to have any hints what might be wrong (and how to resolve it) with my elasticsearch cluster. I've cluster with three master nodes and two data nodes. Most of the indices are configured with couple shards and one replication. Both data nodes have 8 VCPU and 16GB of memory. OS is Debian GNU/Linux 11 (bullseye) Elasticsearch version 7.5.0 Everything (cluster status is green) starts ok, stays like that for couple hours and suddenly one specific data node gets in trouble. Cluster health changes to yellow. From this specific node logs, I see following logs: ```auto [2023-01-27T02:03:05,265][INFO][o.e.i.s.TransportNodesListShardStoreMetaData] [elastic4.mydomain.tld] [myindex-2023.01][1]: failed to obtain shard lock org.elasticsearch.env.ShardLockObtainFailedException: [myindex-2023.01][1]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [read metadata snapshot] at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:769) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:684) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:449) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:171) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:118) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:66) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) [x-pack-security-7.5.0.jar:7.5.0] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] [2023-01-27T02:03:05,327][INFO][o.e.i.s.TransportNodesListShardStoreMetaData] [elastic4.mydomain.tld] [myindex-2023.01][0]: failed to obtain shard lock org.elasticsearch.env.ShardLockObtainFailedException: [myindex-2023.01][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [read metadata snapshot] at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:769) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:684) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:449) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:171) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:118) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:66) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:129) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:244) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:240) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) [x-pack-security-7.5.0.jar:7.5.0] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.0.jar:7.5.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] [2023-01-27T02:03:10,314][WARN][o.e.i.c.IndicesClusterStateService] [elastic4.mydomain.tld] [myindex-2023.01][1] marking and sending shard failed due to [failed to create shard] java.io.IOException: failed to obtain in-memory shard lock at org.elasticsearch.index.IndexService.createShard(IndexService.java:446) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:658) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:165) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:610) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:586) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:266) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.5.0.jar:7.5.0] at java.lang.Iterable.forEach(Iterable.java:75) [?:?] at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:514) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:485) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.access$100(ClusterApplierService.java:73) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:176) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.0.jar:7.5.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] Caused by: org.elasticsearch.env.ShardLockObtainFailedException: [myindex-2023.01][1]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation] at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:769) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:684) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.IndexService.createShard(IndexService.java:366) ~[elasticsearch-7.5.0.jar:7.5.0] ... 18 more [2023-01-27T02:03:15,356][WARN][o.e.i.c.IndicesClusterStateService] [elastic4.mydomain.tld] [myindex-2023.01][0] marking and sending shard failed due to [failed to create shard] java.io.IOException: failed to obtain in-memory shard lock at org.elasticsearch.index.IndexService.createShard(IndexService.java:446) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:658) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:165) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:610) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:586) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:266) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$5(ClusterApplierService.java:517) [elasticsearch-7.5.0.jar:7.5.0] at java.lang.Iterable.forEach(Iterable.java:75) [?:?] at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:514) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:485) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService.access$100(ClusterApplierService.java:73) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:176) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.0.jar:7.5.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] Caused by: org.elasticsearch.env.ShardLockObtainFailedException: [myindex-2023.01][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation] at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:769) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:684) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.IndexService.createShard(IndexService.java:366) ~[elasticsearch-7.5.0.jar:7.5.0] ... 18 more ``` When I read logs from the active master node, it shows "node not connected" errors and shard lock errors from the data node (elastic4.mydomain.tld): ```auto [2023-01-27T02:03:00,193][WARN][o.e.c.r.a.AllocationService] [elastic-master2.mydomain.tld] failing shard [failed shard, shard [myindex-2023.01][1], node[YfkLcj6FS4iZlvUsen_s2A], [R], s[STARTED], a[id=zJ6424jgSJqMjzXZQdYIZg], message [failed to perform indices:data/write/bulk[s] on replica [myindex-2023.01][1], node[YfkLcj6FS4iZlvUsen_s2A ], [R], s[STARTED], a[id=zJ6424jgSJqMjzXZQdYIZg]], failure [NodeNotConnectedException[[elastic4.mydomain.tld][10.13.37.93:9300] Node not connected]], markAsStale [true]] org.elasticsearch.transport.NodeNotConnectedException: [elastic4.mydomain.tld][10.13.37.93:9300] Node not connected at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:189) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:617) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:589) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1035) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:173) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:160) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:135) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:285) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:188) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:170) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:193) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:118) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:917) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:108) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:394) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:316) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$21(IndexShard.java:2752) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2726) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:858) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:312) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:275) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] [2023-01-27T02:03:00,199][INFO][o.e.c.r.a.AllocationService] [elastic-master2.mydomain.tld] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[myindex-2023.01][1]]]). [2023-01-27T02:03:00,256][WARN][o.e.c.r.a.AllocationService] [elastic-master2.mydomain.tld] failing shard [failed shard, shard [myindex-2023.01][0], node[YfkLcj6FS4iZlvUsen_s2A], [R], s[STARTED], a[id=s7M2kw_mSP6oDNr-7NgCMA], message [failed to perform indices:data/write/bulk[s] on replica [myindex-2023.01][0], node[YfkLcj6FS4iZlvUsen_s2A ], [R], s[STARTED], a[id=s7M2kw_mSP6oDNr-7NgCMA]], failure [NodeNotConnectedException[[elastic4.mydomain.tld][10.13.37.93:9300] Node not connected]], markAsStale [true]] org.elasticsearch.transport.NodeNotConnectedException: [elastic4.mydomain.tld][10.13.37.93:9300] Node not connected at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:189) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:617) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:589) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1035) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:173) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:160) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:135) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:285) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:188) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:170) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:193) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:118) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:917) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:108) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:394) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:316) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$21(IndexShard.java:2752) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2726) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:858) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:312) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:275) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:752) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.0.jar:7.5.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.0.jar:7.5.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:830) [?:?] ``` When I query for cluster allocation `http://elastic-master2.mydomain.tld:9200/_cluster/allocation/explain?pretty` information, I get following: ```auto { "index": "myindex-2023.01", "shard": 0, "primary": false, "current_state": "unassigned", "unassigned_info": { "reason": "ALLOCATION_FAILED", "at": "2023-01-27T02:03:48.572Z", "failed_allocation_attempts": 5, "details": "failed shard on node [YfkLcj6FS4iZlvUsen_s2A]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[myindex-2023.01][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ", "last_allocation_status": "no_attempt" }, "can_allocate": "no", "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions": [{ "node_id": "-5YVQVTOQNOzVm2dC6IMhw", "node_name": "elastic3.mydomain.tld", "transport_address": "10.13.37.23:9300", "node_attributes": { "ml.machine_memory": "16760512512", "ml.max_open_jobs": "20", "xpack.installed": "true" }, "node_decision": "no", "deciders": [{ "decider": "max_retry", "decision": "NO", "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2023-01-27T02:03:48.572Z], failed_attempts[5], failed_nodes[[YfkLcj6FS4iZlvUsen_s2A]], delayed=false, details[failed shard on node [YfkLcj6FS4iZlvUsen_s2A]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[myindex-2023.01][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ], allocation_status[no_attempt]]]" }, { "decider": "same_shard", "decision": "NO", "explanation": "the shard cannot be allocated to the same node on which a copy of the shard already exists [[myindex-2023.01][0], node[-5YVQVTOQNOzVm2dC6IMhw], [P], s[STARTED], a[id=78cNqX_VTfai09zPS6B0Tg]]" } ] }, { "node_id": "YfkLcj6FS4iZlvUsen_s2A", "node_name": "elastic4.mydomain.tld", "transport_address": "10.13.37.93:9300", "node_attributes": { "ml.machine_memory": "16761794560", "ml.max_open_jobs": "20", "xpack.installed": "true" }, "node_decision": "no", "deciders": [{ "decider": "max_retry", "decision": "NO", "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2023-01-27T02:03:48.572Z], failed_attempts[5], failed_nodes[[YfkLcj6FS4iZlvUsen_s2A]], delayed=false, details[failed shard on node [YfkLcj6FS4iZlvUsen_s2A]: failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[myindex-2023.01][0]: obtaining shard lock timed out after 5000ms, previous lock details: [shard creation] trying to lock for [shard creation]]; ], allocation_status[no_attempt]]]" }] } ] } ``` Any ideas? Please let me know if you require more information. Many thanks in advance! Br, Aki ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 27, 2023, 8:29am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/2 "2023-01-27T08:29:11Z")
Elasticsearch version 7.5 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 27, 2023, 8:42am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/3 "2023-01-27T08:42:30Z")
What type of storage are you using? What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/cluster-stats.html)? Note that the version you are using is very old and EOL. I would recommend upgrading to at least the latest in the 7.17 line. ---
**Author:** ![akihu](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@akihu](https://discuss.elastic.co/u/akihu)\ **Post date:** [January 27, 2023, 10:14am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/4 "2023-01-27T10:14:47Z")
Hey Christian, First of all thanks for your response, appreciated! All nodes are using local storage (SSD) directly from virtualization host. Cluster stats looks like this: ```auto { "_nodes": { "total": 5, "successful": 5, "failed": 0 }, "cluster_name": "mycluster-2018", "cluster_uuid": "jk0W8Wz0SBmzMvb2-O1CVQ", "timestamp": 1674811993634, "status": "yellow", "indices": { "count": 122, "shards": { "total": 381, "primaries": 193, "replication": 0.9740932642487047, "index": { "shards": { "min": 2, "max": 10, "avg": 3.122950819672131 }, "primaries": { "min": 1, "max": 5, "avg": 1.5819672131147542 }, "replication": { "min": 0, "max": 1, "avg": 0.9877049180327869 } } }, "docs": { "count": 808055906, "deleted": 12052885 }, "store": { "size_in_bytes": 883266203149 }, "fielddata": { "memory_size_in_bytes": 0, "evictions": 0 }, "query_cache": { "memory_size_in_bytes": 9239967, "total_count": 240064, "hit_count": 3389, "miss_count": 236675, "cache_size": 447, "cache_count": 447, "evictions": 0 }, "completion": { "size_in_bytes": 0 }, "segments": { "count": 5112, "memory_in_bytes": 789364946, "terms_memory_in_bytes": 420509570, "stored_fields_memory_in_bytes": 299113552, "term_vectors_memory_in_bytes": 0, "norms_memory_in_bytes": 3877696, "points_memory_in_bytes": 61217870, "doc_values_memory_in_bytes": 4646258, "index_writer_memory_in_bytes": 281553042, "version_map_memory_in_bytes": 31277150, "fixed_bit_set_memory_in_bytes": 6928, "max_unsafe_auto_id_timestamp": 1674808965479, "file_sizes": {} } }, "nodes": { "count": { "total": 5, "coordinating_only": 0, "data": 2, "ingest": 2, "master": 3, "ml": 2, "voting_only": 0 }, "versions": [ "7.5.0" ], "os": { "available_processors": 36, "allocated_processors": 36, "names": [{ "name": "Linux", "count": 5 }], "pretty_names": [{ "pretty_name": "Debian GNU/Linux 11 (bullseye)", "count": 2 }, { "pretty_name": "Debian GNU/Linux 9 (stretch)", "count": 3 } ], "mem": { "total_in_bytes": 67060211712, "free_in_bytes": 15824715776, "used_in_bytes": 51235495936, "free_percent": 24, "used_percent": 76 } }, "process": { "cpu": { "percent": 3 }, "open_file_descriptors": { "min": 338, "max": 7801, "avg": 2977 } }, "jvm": { "max_uptime_in_millis": 113766446, "versions": [{ "version": "13.0.1", "vm_name": "OpenJDK 64-Bit Server VM", "vm_version": "13.0.1+9", "vm_vendor": "AdoptOpenJDK", "bundled_jdk": true, "using_bundled_jdk": true, "count": 5 }], "mem": { "heap_used_in_bytes": 8374540600, "heap_max_in_bytes": 29768351744 }, "threads": 450 }, "fs": { "total_in_bytes": 6435828801536, "free_in_bytes": 5541321633792, "available_in_bytes": 5214115545088 }, "plugins": [], "network_types": { "transport_types": { "security4": 5 }, "http_types": { "security4": 5 } }, "discovery_types": { "zen": 5 }, "packaging_types": [{ "flavor": "default", "type": "deb", "count": 5 }] } } ``` I've noticed the same regarding my cluster version, but before running to upgrade, I was hoping to get somekind understanding about this issue. Br, Aki ---
**Author:** ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee)\ **Post date:** [January 29, 2023, 4:16am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/5 "2023-01-29T04:16:21Z")
Your open file descriptors is very low. should be set to 65,536 or higher. > **[File Descriptors | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html)** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 29, 2023, 5:00am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/6 "2023-01-29T05:00:39Z")
> [@Sunile\_Manjee](#): > > Your open file descriptors is very low. should be set to 65,536 or higher. That's not correct. You're looking at the maximum number of _actually open_ FDs which is not something the user can control. If the _limit_ is wrong then Elasticsearch won't even start. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 29, 2023, 5:14am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/7 "2023-01-29T05:14:46Z")
> [@akihu](#): > > before running to upgrade, I was hoping to get somekind understanding about this issue. This is the wrong way round IMO. We've definitely made newer versions more robust, but also we've made them easier to troubleshoot. Upgrade first, and then if you're still having problems we will find it much easier to help. ---
**Author:** ![akihu](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@akihu](https://discuss.elastic.co/u/akihu)\ **Post date:** [January 29, 2023, 10:19am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/8 "2023-01-29T10:19:31Z")
Thanks David, I will follow yours and Christians suggestion for version upgrade. Should I follow [Rolling upgrades | Elasticsearch Guide [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/rolling-upgrades.html) guide completely or can I only upgrade this specific node that gives me trouble? Br, Aki ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 29, 2023, 10:22am UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/9 "2023-01-29T10:22:09Z")
All nodes in the cluster need to run exactly the same version. ---
**Author:** ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee)\ **Post date:** [January 29, 2023, 2:00pm UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/10 "2023-01-29T14:00:36Z")
Ah yes. My bad. I read that wrong. The correct way to find out is: curl -X GET "localhost:9200/\_nodes/stats/process?filter\_path=\*\*.max\_file\_descriptors&pretty" ---
**Author:** ![akihu](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@akihu](https://discuss.elastic.co/u/akihu)\ **Post date:** [January 29, 2023, 10:49pm UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/11 "2023-01-29T22:49:33Z")
FYI, I've upgraded the whole cluster to version 7.17.8. Before upgrading, I checked the open file descriptors as suggested by Sunile and all nodes returned 65536 (value is the same after upgrade): ```auto { "nodes": { "uHVsT15iTzyFO5QvY2JafA": { "process": { "max_file_descriptors": 65535 } }, "sJFbC5ZySM-ystMzKwTj_w": { "process": { "max_file_descriptors": 65535 } }, "-5YVQVTOQNOzVm2dC6IMhw": { "process": { "max_file_descriptors": 65535 } }, "ixLLmEr4QqCazPcY4CH3pQ": { "process": { "max_file_descriptors": 65535 } }, "YfkLcj6FS4iZlvUsen_s2A": { "process": { "max_file_descriptors": 65535 } } } } ``` Br, Aki ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2023, 10:50pm UTC](https://discuss.elastic.co/t/shard-lock-issue/324094/12 "2023-02-26T22:50:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.