LockReleaseFailedException

Elastic Stack Elasticsearch
1

Hi,
I got this exception in our production servers and the cluster went down.
Any idea what might have caused it and how to prevent this from happening
in the future?

[2013-11-22 03:17:57,600][WARN ][cluster.action.shard ]
[savvy_es_chef_node_1] sending failed shard for [events_addon_v3][4],
node[LYjH3PdYTh6WVE2hL5K0zQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[events_addon_v3][4]
failed recovery]; nested:
EngineCreationFailureException[[events_addon_v3][4] failed to create
engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a
NativeFSLock which is held by another indexer component:
/mnt/elasticsearch/data/savvy_es_prod/nodes/0/indices/events_addon_v3/4/index/write.lock];
]]

--
Best Regards,

Ram Viswanadha

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey,

do you run another elasticsearch instance on the same data, which was not
stoppde (use 'jps' command to see running java instances).
Is the /mnt mountpoint a network share, maybe NFS?
You could also use 'lsof' command to find out, which process has opened the
write.lock file.

--Alex

On Fri, Nov 22, 2013 at 6:29 AM, Ram Viswanadha rviswanadha@gmail.comwrote:

Hi,
I got this exception in our production servers and the cluster went down.
Any idea what might have caused it and how to prevent this from happening
in the future?

[2013-11-22 03:17:57,600][WARN ][cluster.action.shard ]
[savvy_es_chef_node_1] sending failed shard for [events_addon_v3][4],
node[LYjH3PdYTh6WVE2hL5K0zQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[events_addon_v3][4]
failed recovery]; nested:
EngineCreationFailureException[[events_addon_v3][4] failed to create
engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a
NativeFSLock which is held by another indexer component:
/mnt/elasticsearch/data/savvy_es_prod/nodes/0/indices/events_addon_v3/4/index/write.lock];
]]

--
Best Regards,

Ram Viswanadha

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Alex,

do you run another elasticsearch instance on the same data, which was not
stoppde (use 'jps' command to see running java instances).

No
ubuntu@ip1:~$ jps
28983 Jps

ubuntu@ip2:~$ jps
25772 Jps

Is the /mnt mountpoint a network share, maybe NFS?

No. /mnt is IO optimized EBS volume.

You could also use 'lsof' command to find out, which process has opened
the write.lock file.

will try that if it happens next time

--Alex

On Fri, Nov 22, 2013 at 6:29 AM, Ram Viswanadha <rvisw...@gmail.com<javascript:>

wrote:

Hi,
I got this exception in our production servers and the cluster went down.
Any idea what might have caused it and how to prevent this from happening
in the future?

[2013-11-22 03:17:57,600][WARN ][cluster.action.shard ]
[savvy_es_chef_node_1] sending failed shard for [events_addon_v3][4],
node[LYjH3PdYTh6WVE2hL5K0zQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[events_addon_v3][4]
failed recovery]; nested:
EngineCreationFailureException[[events_addon_v3][4] failed to create
engine]; nested: LockReleaseFailedException[Cannot forcefully unlock a
NativeFSLock which is held by another indexer component:
/mnt/elasticsearch/data/savvy_es_prod/nodes/0/indices/events_addon_v3/4/index/write.lock];
]]

--
Best Regards,

Ram Viswanadha

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.