Hi all,
We have a production environment on ElasticSearch 1.6.0.
When we start elasticsearch service, although service status is "started", it does not bind to ports 9200/9300 and join the cluster.
When I check the log, I see that the node initialization process is stuck at "[node] initializing ..."state. It does not proceed to [node] "initialized", "starting ..." etc. states . I pasted the log at the very below.
Thinking indices are corrupt, I stopped the service, renamed the folder where indices are stored and restarted the service. A new folder was created, but it was blank and the same problem continued.
One thing I noticed is that, the java process keeps running at 100% on one of the CPU cores, without any changes for hours, even days.
I am absolutely stuck as the service is not even bound to 9200 port to check its health via curl, and there is no error message on logs. There was not a single configuration change, no network connectivity issues, it has been running stable for several years, only the VM it was hosted was restarted (with graceful node shutdown afaik).
Any help will be greatly appreciated! Thanks!
Note: A fresh installation or upgrade if not possible as the rest of the cluster is up and running.
[2020-01-25 16:31:27,064][INFO ][node] [sk-node-3] version[1.6.0], pid[1253], b uild[cdd3ac4/2015-06-09T13:36:34Z]
[2020-01-25 16:31:27,120][INFO ][node] [sk-node-3] initializing ...
[2020-01-25 16:31:27,210][INFO ][plugins] [sk-node-3] loaded , sites [head, bigd esk]
[2020-01-25 16:31:27,576][INFO ][env] [sk-node-3] using [1] data paths, mounts [[/datadrive2 (/dev/sdc1)]], net usable_space [26.9gb], net total_space [29.4gb], types [ext4]
[2020-01-25 16:31:31,786][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/calculate-score.groovy]
[2020-01-25 16:31:32,473][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/predict-item-score-5-latent.groovy]
[2020-01-25 16:31:32,539][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/predict-item-score-6-latent.groovy]
[2020-01-25 16:31:32,649][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/predict-item-score.groovy]
[2020-01-25 16:31:32,670][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/predict-item-score2.groovy]
[2020-01-25 16:31:32,697][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/predict-item-score3.groovy]
[2020-01-25 16:31:32,723][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/predict-item-score4.groovy]
[2020-01-25 16:31:32,767][INFO ][script] [sk-node-3] compiling script file [/etc/ elasticsearch/scripts/user-similarity.groovy]
--- END OF LOG---
