# Underlying file changed by an external force

**URL:** <https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/258648>\
**Category:** Elasticsearch\
**Created:** [December 14, 2020, 9:26pm UTC](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/258648 "2020-12-14T21:26:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [December 14, 2020, 9:26pm UTC](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/258648/1 "2020-12-14T21:26:46Z")

</div>

Hi Team,

Of late, I've started seeing these errors in ES 6.8.6 cluster. I have this cluster since March but these errors started appearing only recently:

```
[2020-12-14T00:00:04,122][WARN][o.e.c.r.a.AllocationService] [node1.foo.bar.com] 
failing shard [failed shard, shard [.kibana_7][0], node[9b3APiVrTliXlxUA4RR3Rg], [R],
s[STARTED], a[id=EwFxAm38RkiMwEgbV7bGaA], message [failed to perform
indices:data/write/bulk[s] on replica [.kibana_7][0], node[9b3APiVrTliXlxUA4RR3Rg], [R],
s[STARTED], a[id=EwFxAm38RkiMwEgbV7bGaA]], failure 
[RemoteTransportException[[node1.foo.bar.com][1.1.1.1:9300][indices:data/write/bulk[s][r]]]; 
nested: AlreadyClosedException[Underlying file changed by an external force at 2020-12-
10T05:58:11Z, 
   (lock=NativeFSLock(path=/data/disk1/data/nodes/0/indices/46Z0vkIURCCtFLIKx3aHow/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 
exclusive valid],creationTime=2020-12-10T05:58:11.385213Z))]; ], markAsStale [true]]
org.elasticsearch.transport.RemoteTransportException: [node1.foo.bar.com][1.1.1.1:9300][indices:data/write/bulk[s][r]]

```

This happens once a week or more often as well. I initially suspected the Qualys scan agent and had it disabled but still the error appears. The cluster goes into yellow state but automatically recovers on its own without me having to restart the ES Cluster. Sometimes, though I do have to restart the cluster or else just close and re-open the affected index.

Can anyone shed light on what could be wrong? Is there a way to know which process is modifying the files? I suspect it could be Anti-virus since the UUID names as folder might be suspicious to it. But how can I know which process modifies it?

```
[root@node1.foo.bar.com]# ll -lrt
total 212
-rw-r--r--. 1 elasticsearch elasticsearch 0 Dec 14 00:00 write.lock

```

**ES Cluster** : 6.8.6. Self managed.  
**Data Node** : 55 GB RAM. 8 TB SSDs. 16 cores.

Total 10 data nodes and 3 master nodes.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 14, 2020, 10:53pm UTC](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/258648/2 "2020-12-14T22:53:33Z")

</div>

> [@sandeepkanabar](#):
>
> Can anyone shed light on what could be wrong? Is there a way to know which process is modifying the files? I suspect it could be Anti-virus since the UUID names as folder might be suspicious to it. But how can I know which process modifies it?

It's definitely something other than Elasticsearch meddling with Elasticsearch's data. it could well be something like an antivirus program. Pinning down the specific process that's causing your problems is tricky, however, particularly if you've tried disabling the suspects without success. You could try running `lsof` in a loop in the hope of catching another process looking at Elasticsearch's files?

---

<div class="post-metadata">

**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\
**Post date:** [December 17, 2020, 12:13pm UTC](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/258648/3 "2020-12-17T12:13:10Z")

</div>

Thanks David. I set-up the following audit rule:

```
auditctl -a always,exit -F dir=/data/disk1/data/nodes/0/indices -F perm=wa -F uid!=elasticsearch -k mykey

```

This will monitor any files changes (`wa` - writes and attribute changes) in the directory `/data/disk1/data/nodes/0/indices` and its sub-dirs. And it will `exclude` the changes made by `elasticsearch` user.

And then set-up a cron job that checks if `/sbin/ausearch -i --input-logs -k mykey` has any output and if so, triggers an email alert.

Let me know if you have any other suggestions.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 14, 2021, 12:13pm UTC](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/258648/4 "2021-01-14T12:13:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.