ElasticSearch failure: Environment is not locked

Elastic Stack Elasticsearch
1

Hi,

Environment:
OS : Windows 64-bit
ElasticSearch : 6.6.2
Java: 1.8.151
Cluster: Single Node cluster
Index strategy: Indices are created daily and programmatically
Shard: Single shard per index and no replication

We have been using elasticsearch for more than a month.
The machine contains approximately 30 indices (data for 30 days).
Total size of all the indices is around 100s of GBs.

[2019-04-19T10:28:40,735][WARN ][o.e.e.NodeEnvironment    ] [Heqvpo8] lock assertion failed
java.io.IOException: The device is not ready
	at sun.nio.ch.FileDispatcherImpl.size0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.size(FileDispatcherImpl.java:97) ~[?:?]
	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:310) ~[?:?]
	at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:182) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
	at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:999) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.env.NodeEnvironment.indexPaths(NodeEnvironment.java:798) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:125) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.gateway.GatewayMetaState.applyClusterState(GatewayMetaState.java:176) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:486) ~[elasticsearch-6.6.2.jar:6.6.2]
	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_151]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:483) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:470) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:421) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:165) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.6.2.jar:6.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2019-04-19T10:28:40,735][WARN ][o.e.e.NodeEnvironment    ] [Heqvpo8] lock assertion failed
java.io.IOException: The device is not ready
	at sun.nio.ch.FileDispatcherImpl.size0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.size(FileDispatcherImpl.java:97) ~[?:?]
	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:310) ~[?:?]
	at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:182) ~[lucene-core-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f - nknize - 2018-12-07 14:44:20]
	at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:999) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.env.NodeEnvironment.availableShardPaths(NodeEnvironment.java:817) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:120) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) [x-pack-security-6.6.2.jar:6.6.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) [x-pack-security-6.6.2.jar:6.6.2]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:687) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2019-04-19T10:28:40,739][WARN ][o.e.g.MetaStateService   ] [Heqvpo8] [[ktinspectorlogs-18-04-2019/0mW-XRLvSneko2uxm2cGWg]]: failed to write index state
java.lang.IllegalStateException: environment is not locked
	at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1002) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.env.NodeEnvironment.indexPaths(NodeEnvironment.java:798) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.gateway.MetaStateService.writeIndex(MetaStateService.java:125) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.gateway.GatewayMetaState.applyClusterState(GatewayMetaState.java:176) ~[elasticsearch-6.6.2.jar:6.6.2]
	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:486) ~[elasticsearch-6.6.2.jar:6.6.2]
	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_151]

Caused by: java.lang.IllegalStateException: environment is not locked
2

This issue happens every time I restart the machine.
The workaround is to restart elasticsearch service.

What is the root cause of this error?

3

It looks like you're starting up Elasticsearch before the storage device is ready.

4

The service is running in Auto mode (Windows elastic service). So it tries to start on system boot.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.