Health check failed

Elastic Stack Elasticsearch
#1

I am trying to install Elasticserver in Linux, after I ran the server I am facing the below issue. Can you please help me to resolve this?

[2020-12-22T00:02:31,363][ERROR][o.e.m.f.FsHealthService ] [GTHYJDEV01.pneus.touchette] health check failed
org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-12-20T20:46:05Z, (lock=NativeFSLock(path=/opt/ELK/elasticsearch-7.10.1/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-12-19T18:05:27Z))
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1044) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:803) ~[elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:158) [elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:146) [elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:213) [elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.1.jar:7.10.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_271]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_271]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_271]

#2

Something (not Elasticsearch) is meddling with the contents of /opt/ELK/elasticsearch-7.10.1/data. This is not permitted since it can lead to corruption and data loss. You must make sure that only Elasticsearch is allowed access to this path.

#3

Hi David,

Thanks for your reply

I am trying to understand the Error which you mentioned above but i have give the path access to each folder (read, write and exec) permission.

but still facing the same issue.

Thanks,
Suresh Koganti.

#4

Did you perhaps run Elasticsearch more than once, without quitting the previous one?

In Linux you can try ps -x | grep elasticsearch and check if more than one instance is up.

#5

It can't be that, Elasticsearch uses advisory file locks to prevent it, and never modifies the file that is being modified here anyway.

Something (not Elasticsearch) is modifying the files on which Elasticsearch depends. You must stop this from happening.

#6

Hi David,

Here is info after i ran the ps -x | grep elasticsearch

[skkoganti@GTHYJDEV01 bin] ps -x | grep elasticsearch 104881 ? Sl 36:07 /usr/jdk/jdk1.8.0_271/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,JRE -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/elasticsearch-4071042967598299053 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:MaxDirectMemorySize=536870912 -Des.path.home=/opt/ELK/elasticsearch-7.10.1 -Des.path.conf=/opt/ELK/elasticsearch-7.10.1/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /opt/ELK/elasticsearch-7.10.1/lib/* org.elasticsearch.bootstrap.Elasticsearch 105043 ? Sl 0:00 /opt/ELK/elasticsearch-7.10.1/modules/x-pack-ml/platform/linux-x86_64/bin/controller 108760 ? Sl 32:19 /opt/ELK/elasticsearch-7.10.1/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-805487726459299218 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=536870912 -Des.path.home=/opt/ELK/elasticsearch-7.10.1 -Des.path.conf=/opt/ELK/elasticsearch-7.10.1/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /opt/ELK/elasticsearch-7.10.1/lib/* org.elasticsearch.bootstrap.Elasticsearch 108947 ? Sl 0:00 /opt/ELK/elasticsearch-7.10.1/modules/x-pack-ml/platform/linux-x86_64/bin/controller 130948 pts/1 S+ 0:00 grep --color=auto elasticsearch [skkoganti@GTHYJDEV01 bin]

#7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.