org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force

[rdl (roberto)]

Environment:

• GlusterFS 6.0
• OpenShift 3.11
• ES 6

Hi,
while deploying sonarqube which in turn uses ES6 in a POD I encounter the following error:

2020.01.16 12:16:19 WARN es[o.e.c.r.a.AllocationService] failing shard [failed shard, shard [projectmeasures][4], node[yj7F4GFeTLWJIBBGdlG6OA], [P], s[STARTED], a[id=PYKYQXW4S26dAErK2wBVCQ], message [shard failure, reason [lucene commit failed]], failure [AlreadyClosedException[Underlying file changed by an external force at 2020-01-16T09:50:02.846963Z, (lock=NativeFSLock(path=/opt/sonarqube/data/es6/nodes/0/indices/r1Iw474jQPWfDDwVoJoAwg/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-01-16T09:50:02.846628Z))]], markAsStale [true]]
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-01-16T09:50:02.846963Z, (lock=NativeFSLock(path=/opt/sonarqube/data/es6/nodes/0/indices/r1Iw474jQPWfDDwVoJoAwg/4/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-01-16T09:50:02.846628Z))

Do you have any suggestions?

Thanks in advance
Bests

[DavidTurner]

Looks similar to this post:

Underlying file changed by an external force

The first two Google hits for elasticsearch glusterfs lead to two serious-looking open GlusterFS issues that Elasticsearch triggers: https://bugzilla.redhat.com/show_bug.cgi?id=1390050 https://bugzilla.redhat.com/show_bug.cgi?id=1379568 This may be one of them, or may be something else, but it seems worth rethinking your setup. I don't think so. Both of the times I quoted are in UTC (note the trailing Z) and differ by 28 seconds, so it seems unlikely to be a timezone issue.

In particular, both involve GlusterFS. Try using a local filesystem instead.

[rdl (roberto)]

Hi David, thank you for your suggestion. I already saw that post and investigated glusterFS bugs triggered by ES, specifically the one related to the ctime (https://bugzilla.redhat.com/show_bug.cgi?id=1318493). This bug has been solved (https://review.gluster.org/#/c/glusterfs/+/19857/) since the glusterFS version 5. I also checked the timezone and it is the same.

[DavidTurner]

@rdl

This bug has been solved

That bug looks related but not completely relevant since the times in question are, I believe, the mtime of the write.lock file. Elasticsearch is reporting that its mtime is changing underneath it despite nothing modifying it. It interprets this to mean that it doesn't have exclusive control over its data path and shuts down to protect itself.