org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-05-15T07:17:37Z

Elastic Stack Elasticsearch
1

Dear Team,

Could you please help us on the below error.

[2018-04-04T15:01:38,250][WARN ][o.e.c.s.ClusterApplierService] [node_3] failed to notify ClusterStateListener
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-05-15T07:17:37Z, (lock=NativeFSLock(path=/opt/app/elasticsearch/data_1/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-04-04T06:45:44Z))

Steps to Recreate

i. Stop elasticsearch node
ii. Comment below lines in the /etc/fstab
/dev/HKLPADEDM26_applvgsda/es_data /opt/app/elasticsearch/data_1 ext4 defaults 0 0
/dev/HKLPADEDM26_applvgsdb/es_data /opt/app/elasticsearch/data_2 ext4 defaults 0 0
/dev/HKLPADEDM26_applvgsdc/es_data /opt/app/elasticsearch/data_3 ext4 defaults 0 0
/dev/HKLPADEDM26_applvgsdd/es_data /opt/app/elasticsearch/data_4 ext4 defaults 0 0

iii. Note the inode number, Create time and Modify time for node.lock
Inode Number: 17301507, Modify time: Mon May 15 15:17:37 2017, Creation time: Mon May 15 15:17:37 2017
iv. Reboot the server and note the inode number, Create time and Modify time for node.lock
Inode Number: 524986, Modify time: Wed Apr 4 14:45:44 2018, Creation time: Wed Apr 4 14:45:44 2018
v. Uncomment the FS in /etc/fstab and Mount the Data Filesystem.
Note the inode number, Create time and Modify time for node.lock
Inode Number: 17301507, Modify time: Mon May 15 15:17:37 2017, Creation time: Mon May 15 15:17:37 2017
vi. After that getting below errors.

org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2017-05-15T07:17:37Z, (lock=NativeFSLock(path=/opt/app/elasticsearch/data_1/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2018-04-04T06:45:44Z))

2

This error means that the node.lock file's creation time has changed since Elasticsearch started up, which indicates that something fundamental is going wrong in the environment around Elasticsearch.

I believe you are running Elasticsearch at step (iv) and then mounting a different filesystem underneath it at step (v) while it is still running. This completely changes the files available to Elasticsearch, which would explain this message. It is very important to make sure you have completed mounting your filesystems before you start up Elasticsearch.

3

Thanks David.

Still one thing I am wondering, At sometime it recovered automatically.

4

Elasticsearch will continue to report this error until either it is restarted or until it finds the node.lock file to have the correct creation time. Neither of these things happen automatically.

I believe your system recovered because you restarted Elasticsearch with the data filesystem correctly mounted.

5

Agreed. Restart make sense here and It solved the issue.
Thanks David. :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.