org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force

Elastic Stack Elasticsearch
1

Environment:

  • GlusterFS 6.0
  • OpenShift 3.11
  • ES 6

Hi,
while deploying sonarqube which in turn uses ES6 in a POD I encounter the following error:

2020.01.16 12:16:19 WARN es[o.e.c.r.a.AllocationService] failing shard [failed shard, shard [projectmeasures][4], node[yj7F4GFeTLWJIBBGdlG6OA], [P], s[STARTED], a[id=PYKYQXW4S26dAErK2wBVCQ], message [shard failure, reason [lucene commit failed]], failure [AlreadyClosedException[Underlying file changed by an external force at 2020-01-16T09:50:02.846963Z, (lock=NativeFSLock(path=/opt/sonarqube/data/es6/nodes/0/indices/r1Iw474jQPWfDDwVoJoAwg/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-01-16T09:50:02.846628Z))]], markAsStale [true]]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-01-16T09:50:02.846963Z, (lock=NativeFSLock(path=/opt/sonarqube/data/es6/nodes/0/indices/r1Iw474jQPWfDDwVoJoAwg/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-01-16T09:50:02.846628Z))

Do you have any suggestions?

Thanks in advance
Bests

2

Looks similar to this post:

In particular, both involve GlusterFS. Try using a local filesystem instead.

3

Hi David, thank you for your suggestion. I already saw that post and investigated glusterFS bugs triggered by ES, specifically the one related to the ctime (https://bugzilla.redhat.com/show_bug.cgi?id=1318493). This bug has been solved (https://review.gluster.org/#/c/glusterfs/+/19857/) since the glusterFS version 5. I also checked the timezone and it is the same.

4

That bug looks related but not completely relevant since the times in question are, I believe, the mtime of the write.lock file. Elasticsearch is reporting that its mtime is changing underneath it despite nothing modifying it. It interprets this to mean that it doesn't have exclusive control over its data path and shuts down to protect itself.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.