# Fatal error in thread exiting **URL:** \ **Category:** Elasticsearch\ **Created:** [March 14, 2020, 12:18pm UTC](https://discuss.elastic.co/t/fatal-error-in-thread-exiting/223614 "2020-03-14T12:18:32Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![kangning1](https://avatars.discourse-cdn.com/v4/letter/k/5e9695/32.png) [@kangning1](https://discuss.elastic.co/u/kangning1)\ **Post date:** [March 14, 2020, 12:18pm UTC](https://discuss.elastic.co/t/fatal-error-in-thread-exiting/223614/1 "2020-03-14T12:18:32Z")
hi: I received an error. When es imported the data, it reported an error and exited, and the cluster stuck. Elasticsearch runs on kubernetes, using the glusterfs store. Cluster information： sh-4.2$ curl 127.0.0.1:9200 { "name" : "elastic-header-1", "cluster\_name" : "es-cluster", "cluster\_uuid" : "53kTWYq7RXuyyqHojyScSQ", "version" : { "number" : "6.4.2", "build\_flavor" : "default", "build\_type" : "tar", "build\_hash" : "04711c2", "build\_date" : "2018-09-26T13:34:09.098244Z", "build\_snapshot" : false, "lucene\_version" : "7.4.0", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } log: [2020-03-13T20:49:55,626][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elastic-header-2.elastic-header.ism-mobile.svc.cluster.local] fatal error in thread [elasticsearch[elastic-header-2.elastic-header.ism-mobile.svc.cluster.local][refresh][T#2]], exiting java.lang.AssertionError: Unexpected AlreadyClosedException at org.elasticsearch.index.engine.InternalEngine.failOnTragicEvent(InternalEngine.java:1838) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1410) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1375) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:880) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:696) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:898) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:808) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.4.2.jar:6.4.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0\_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0\_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_211] Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-03-13T12:53:19Z, (lock=NativeFSLock(path=/data/app/es/data/nodes/0/indices/ZrlFhoykSkGGqOnJLU9n3g/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-13T12:46:27Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:664) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:4983) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexWriter.access$200(IndexWriter.java:211) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexWriter$1.lambda$deleteUnusedFiles$0(IndexWriter.java:355) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5065) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:504) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:288) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:253) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:284) ~[elasticsearch-6.4.2.jar:6.4.2] at org.elasticsearch.index.engine.InternalEngine$ExternalSearcherManager.refreshIfNeeded(InternalEngine.java:259) ~[elasticsearch-6.4.2.jar:6.4.2] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.4.0.jar:7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 2018-06-18 16:51:45] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1396) ~[elasticsearch-6.4.2.jar:6.4.2] ... 10 more ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 15, 2020, 9:50am UTC](https://discuss.elastic.co/t/fatal-error-in-thread-exiting/223614/2 "2020-03-15T09:50:28Z")
> [@kangning1](#): > > `Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-03-13T12:53:19Z, (lock=NativeFSLock(path=/data/app/es/data/nodes/0/indices/ZrlFhoykSkGGqOnJLU9n3g/0/index/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-13T12:46:27Z))` The last-modified time of `/data/app/es/data/nodes/0/indices/ZrlFhoykSkGGqOnJLU9n3g/0/index/write.lock` was changed by something other than Elasticsearch. This indicates that something else is writing to Elasticsearch's data path, which is very very bad. You should work out what this "something else" is and stop it. Elasticsearch treats this as an indication that something is very wrong with the environment and stops writing to disk to avoid any further chance of corrupting your data. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 15, 2020, 9:56am UTC](https://discuss.elastic.co/t/fatal-error-in-thread-exiting/223614/3 "2020-03-15T09:56:31Z")
> [@kangning1](#): > > Elasticsearch runs on kubernetes, using the glusterfs store. Oh, just saw this. GlusterFS is a [risky choice](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/137555/4) IMO and there's [no need for it](https://discuss.elastic.co/t/elasticsearch-support-for-distributed-fs-like-rook-ceph/199410/4). ---
**Author:** ![kangning1](https://avatars.discourse-cdn.com/v4/letter/k/5e9695/32.png) [@kangning1](https://discuss.elastic.co/u/kangning1)\ **Post date:** [March 15, 2020, 10:53am UTC](https://discuss.elastic.co/t/fatal-error-in-thread-exiting/223614/4 "2020-03-15T10:53:24Z")
Thanks for your support. I'll try other storage methods. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2020, 10:53am UTC](https://discuss.elastic.co/t/fatal-error-in-thread-exiting/223614/5 "2020-04-12T10:53:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.