We are trying to stop elastic using the service elasticsearch stop command however not sure how it further calls the kill command as there is no execstop command in the elasticsearch.service file
Also we see the process keeps on running but pid file is removed.
elastic 2596439 1 1 Apr18 ? 04:00:04 /path/shared/pathElasticsearch8-1.0.SNAPSHOT/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-18329529858296199006 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:HeapDumpPath=logs -Xms7913m -Xmx7913m -XX:MaxDirectMemorySize=4150263808 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.distribution.type=tar --module-path /path/config/elasticsearch8/nodes/dev01/lib --add-modules=jdk.net -m org.elasticsearch.server/org.elasticsearch.bootstrap.Elasticsearch
elastic 2601266 1 1 Apr18 ? 04:43:47 /path/config/kibana8/bin/../node/bin/node /path/config/kibana8/bin/../src/cli/dist
checking the service
service pathElasticsearch8 status
Redirecting to /bin/systemctl status pathElasticsearch8.service
● pathElasticsearch8.service - Elasticsearch8
Loaded: loaded (/etc/systemd/system/pathElasticsearch8.service; enabled; vendor preset: disabled)
Active: deactivating (stop-sigterm) since Mon 2023-04-24 17:11:36 BST; 1 weeks 0 days ago
Process: 4154989 ExecStop=/path/config/elasticsearch8/bin/elasticsearch stop (code=exited, status=0/SUCCESS)
Main PID: 2596439 (java)
Tasks: 117 (limit: 101037)
Memory: 8.9G
CGroup: /system.slice/pathElasticsearch8.service
└─2596439 /path/shared/pathElasticsearch8-1.0.SNAPSHOT/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.>
We have to hard kill the process every time can someone please explain the reasonz.
Below is the error during shutdown of the elasticsearch8
[2023-04-24T17:11:36,299][INFO ][o.e.x.w.WatcherService ] [dev01] stopping watch service, reason [shutdown initiated]
[2023-04-24T17:11:36,300][INFO ][o.e.x.w.WatcherLifeCycleService] [dev01] watcher has stopped and shutdown
[2023-04-24T17:11:36,577][INFO ][o.e.c.c.Coordinator ] [dev01] master node [{dev03}{R9mOUc7AQ52Z4tAVNVWbSA}{b8Hc4HjGRyWhnT3YSKqmQw}{dev03}{57.255.232.170}{57.255.232.170:9310}{cdfhlmrstw}] disconnected, restarting discovery
[2023-04-24T17:11:36,584][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dev01] [controller/2596473] [Main.cc@176] ML controller exiting
[2023-04-24T17:11:36,583][ERROR][i.n.u.c.D.rejectedExecution] [dev01] Failed to submit a listener notification task. Event loop shut down?
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "setContextClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
at java.lang.Thread.setContextClassLoader(Thread.java:1525) ~[?:?]
at io.netty.util.concurrent.GlobalEventExecutor$2.run(GlobalEventExecutor.java:226) ~[?:?]
at io.netty.util.concurrent.GlobalEventExecutor$2.run(GlobalEventExecutor.java:223) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
at io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:223) ~[?:?]
at io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:211) ~[?:?]
at io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:205) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1057) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2023-04-24T17:11:36,587][INFO ][o.e.x.m.p.NativeController] [dev01] Native controller process has stopped - no new native processes can be started
[2023-04-24T17:11:46,580][WARN ][o.e.c.c.ClusterFormationFailureHelper] [dev01] master not discovered or elected yet, an election requires at least 2 nodes with i
```
