java.nio.channels.OverlappingFileLockException: null

JJerpseth · October 7, 2017, 2:46am

Please note: I have already tried the solutions I found online which included
1 -- reboot machine
2 -- delete node.lock in my data areas

Node dropped out of the cluster then elasticsearch failed to start with

[2017-10-06T15:25:43,864][INFO ][o.e.n.Node ] [colo1-es6] initializing ...
[2017-10-06T15:25:43,937][ERROR][o.e.b.Bootstrap ] Exception
java.nio.channels.OverlappingFileLockException: null
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) ~[?:?]
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) ~[?:?]
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1108) ~[?:?]
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155) ~[?:1.8.0_131]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:114) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b
65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec964
58315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec964
58315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:226) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.node.Node.(Node.java:264) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.node.Node.(Node.java:244) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:232) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.1.jar:5.5.1]

jasontedor · October 7, 2017, 5:08pm

I’ve seen this before when a user inadvertently had a data path defined twice. Can you check for this possibility (check path.data and default.path.data)? We added detection for this but maybe it’s broken so let’s rule this possibility out before trying to think of others. Also, can you start Elasticsearch with -E logger.org.elasticsearch.env.NodeEnvironment=trace. I hope this will give us everything we need to debug the problem here.

JJerpseth · October 9, 2017, 4:18pm

I checked and only have path.data.

Here is the new log file.

[2017-10-09T09:15:57,015][INFO ][o.e.n.Node ] [colo1-es6] initializing ...
[2017-10-09T09:15:57,040][ERROR][o.e.b.Bootstrap ] Exception
java.nio.channels.OverlappingFileLockException: null
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) ~[?:?]
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) ~[?:?]
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1108) ~[?:?]
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155) ~[?:1.8.0_131]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:114) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:226) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.node.Node.(Node.java:264) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.node.Node.(Node.java:244) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:232) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.5.1.jar:5.5.1]
[2017-10-09T09:15:57,047][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [colo1-es6] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.nio.channels.OverlappingFileLockException
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.5.1.jar:5.5.1]
Caused by: java.nio.channels.OverlappingFileLockException
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) ~[?:?]
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) ~[?:?]
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1108) ~[?:?]
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155) ~[?:1.8.0_131]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:114) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:29:46]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:226) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.node.Node.(Node.java:264) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.node.Node.(Node.java:244) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:232) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:351) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.5.1.jar:5.5.1]
... 6 more

jasontedor · October 9, 2017, 8:47pm

Are you sure that you set the loger setting correctly? There should be a log line like obtaining node lock on if the node was indeed started with logger.org.elasticsearch.env.NodeEnvironment set to trace.

system · November 6, 2017, 8:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.