Elasticsearch installation and configuration issue

Elastic Stack Elasticsearch
1

Hello Elasticsearch forum,

My name is Bogdan Oproescu and I am senior data architect for EasyDO Digital Technologies in Bucharest. I am writing you today, to report an issue with our initial Elasticsearch engine configuration failure this morning, on our Linux CentOS server version 8.1. Here is the error message below:

Step 1: install Elasticsearch:
a. For CentOS we use rpm:
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.1-x86_64.rpm
b. sudo rpm -i elasticsearch-7.8.1-x86_64.rpm
2. There were dependency problems on our Linux server, with a previous version of Elastic installed on this server (now deprecated!):
a. view previous versions of Elasticsearch installed:
[root@ElasticEDT bin]# rpm -qa | grep elastic
elasticsearch-7.6.1-1.x86_64
b. remove previous versions of Elasticsearch:
[root@ElasticEDT bin]# rpm -e elasticsearch-7.6.1-1.x86_64
Stopping elasticsearch service... OK
warning: file /etc/elasticsearch/log4j2.properties: remove failed: No such file or directory
warning: /etc/elasticsearch/jvm.options saved as /etc/elasticsearch/jvm.options.rpmsave
warning: file /etc/elasticsearch/elasticsearch.yml: remove failed: No such file or directory
Deleting log directory... OK
[root@ElasticEDT bin]#
c. install the new version of Elasticsearch:
sudo rpm -i elasticsearch-7.8.1-x86_64.rpm
--> and the output is this:
[root@ElasticEDT bin]# sudo rpm -i elasticsearch-7.8.1-x86_64.rpm
Creating elasticsearch group... OK
Creating elasticsearch user... OK

NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

You can start elasticsearch service by executing

sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
[/usr/lib/tmpfiles.d/elasticsearch.conf:1] Line references path below legacy directory /var/run/, updating /var/run/elasticsearch → /run/elasticsearch; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/libgpod.conf:1] Line references path below legacy directory /var/run/, updating /var/run/libgpod → /run/libgpod; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/mdadm.conf:1] Line references path below legacy directory /var/run/, updating /var/run/mdadm → /run/mdadm; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/radvd.conf:1] Line references path below legacy directory /var/run/, updating /var/run/radvd → /run/radvd; please update the tmpfiles.d/ drop-in file accordingly.
[/usr/lib/tmpfiles.d/subscription-manager.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rhsm → /run/rhsm; please update the tmpfiles.d/ drop-in file accordingly.
[root@ElasticEDT bin]#
3. Therefore our new version of Elasticsearch analytics engine was correctly installed on our ElasticEDT server!
a. Now we try to start this new version of Elasticsearch:
[root@ElasticEDT bin]# sudo service elasticsearch start
Starting elasticsearch (via systemctl): Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[FAILED]
b. To find out more, we can execute this command below:
[root@ElasticEDT elasticsearch]# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-09-01 05:49:18 EEST; 3min 38s ago
Docs: https://www.elastic.co
Process: 2709 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2709 (code=exited, status=1/FAILURE)

Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: at org.elasticsearch.cli.Command.main(Command.java:90)
Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Sep 01 05:49:18 ElasticEDT systemd-entrypoint[2709]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
Sep 01 05:49:18 ElasticEDT systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 01 05:49:18 ElasticEDT systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 01 05:49:18 ElasticEDT systemd[1]: Failed to start Elasticsearch.
[root@ElasticEDT elasticsearch]#

2

Can you share the content of the file /var/log/elasticsearch/elasticsearch.log

1 Like
3

Hi Yassine,
Thanks for your prompt reply: I have checked earlier our CentOS 8.1 Linux Elastic server, and inside this folder /var/log/elasticsearch we only have these files below:
[root@ElasticEDT elasticsearch]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@ElasticEDT elasticsearch]# pwd
/var/log/elasticsearch
[root@ElasticEDT elasticsearch]# ll
total 48
-rw-r--r--. 1 elasticsearch elasticsearch 1356 Sep 1 22:04 gc.log
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:14 gc.log.00
-rw-r--r--. 1 elasticsearch elasticsearch 1331 Sep 1 05:14 gc.log.01
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:25 gc.log.02
-rw-r--r--. 1 elasticsearch elasticsearch 1344 Sep 1 05:25 gc.log.03
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:49 gc.log.04
-rw-r--r--. 1 elasticsearch elasticsearch 1343 Sep 1 05:49 gc.log.05
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:56 gc.log.06
-rw-r--r--. 1 elasticsearch elasticsearch 1344 Sep 1 05:56 gc.log.07
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:57 gc.log.08
-rw-r--r--. 1 elasticsearch elasticsearch 1343 Sep 1 05:57 gc.log.09
-rw-r--r--. 1 elasticsearch elasticsearch 907 Sep 1 22:04 gc.log.10
[root@ElasticEDT elasticsearch]#

so which one of these would you like me to send you?
regards, Bogdan

4

if we sort by modification time, newest first, we obtain this listing of files below:
[root@ElasticEDT elasticsearch]# pwd
/var/log/elasticsearch
[root@ElasticEDT elasticsearch]# ls -lt
total 48
-rw-r--r--. 1 elasticsearch elasticsearch 1356 Sep 1 22:04 gc.log
-rw-r--r--. 1 elasticsearch elasticsearch 907 Sep 1 22:04 gc.log.10
-rw-r--r--. 1 elasticsearch elasticsearch 1343 Sep 1 05:57 gc.log.09
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:57 gc.log.08
-rw-r--r--. 1 elasticsearch elasticsearch 1344 Sep 1 05:56 gc.log.07
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:56 gc.log.06
-rw-r--r--. 1 elasticsearch elasticsearch 1343 Sep 1 05:49 gc.log.05
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:49 gc.log.04
-rw-r--r--. 1 elasticsearch elasticsearch 1344 Sep 1 05:25 gc.log.03
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:25 gc.log.02
-rw-r--r--. 1 elasticsearch elasticsearch 1331 Sep 1 05:14 gc.log.01
-rw-r--r--. 1 elasticsearch elasticsearch 898 Sep 1 05:14 gc.log.00
[root@ElasticEDT elasticsearch]#

5

so it makes sense to send you the first 2 files above, as these seem to be periodically updated. Here you go:

  1. [root@ElasticEDT elasticsearch]# more gc.log
    [2020-09-01T19:04:47.333+0000][12495][gc,heap] Heap region size: 1M
    [2020-09-01T19:04:47.334+0000][12495][gc,heap,coops] Heap address: 0x00000000c0000000, size: 1024 MB, Compressed O
    ops mode: 32-bit
    [2020-09-01T19:04:47.583+0000][12495][gc ] Using G1
    [2020-09-01T19:04:47.584+0000][12495][gc,cds ] Mark closed archive regions in map: [0x00000000fff00000, 0x00
    000000fff7eff8]
    [2020-09-01T19:04:47.584+0000][12495][gc,cds ] Mark open archive regions in map: [0x00000000ffe00000, 0x0000
    0000ffe51ff8]
    [2020-09-01T19:04:47.595+0000][12495][gc ] Periodic GC disabled
    [2020-09-01T19:04:48.032+0000][12495][safepoint ] Safepoint "ICBufferFull", Time since last: 441088403 ns, Reac
    hing safepoint: 169098 ns, At safepoint: 15859 ns, Total: 184957 ns
    [2020-09-01T19:04:48.603+0000][12495][gc,heap,exit ] Heap
    [2020-09-01T19:04:48.603+0000][12495][gc,heap,exit ] garbage-first heap total 1048576K, used 47940K [0x00000000
    c0000000, 0x0000000100000000)
    [2020-09-01T19:04:48.603+0000][12495][gc,heap,exit ] region size 1024K, 46 young (47104K), 0 survivors (0K)
    [2020-09-01T19:04:48.603+0000][12495][gc,heap,exit ] Metaspace used 10816K, capacity 11304K, committed 1152
    0K, reserved 1058816K
    [2020-09-01T19:04:48.603+0000][12495][gc,heap,exit ] class space used 1349K, capacity 1500K, committed 1536K,
    reserved 1048576K
    [root@ElasticEDT elasticsearch]#
  2. and for the second file:
    [root@ElasticEDT elasticsearch]# more gc.log.10
    [2020-09-01T19:04:47.183+0000][12649][gc,heap] Heap region size: 1M
    [2020-09-01T19:04:47.183+0000][12649][gc,heap,coops] Heap address: 0x00000000c0000000, size: 1024 MB, Compressed O
    ops mode: 32-bit
    [2020-09-01T19:04:47.192+0000][12649][gc ] Using G1
    [2020-09-01T19:04:47.236+0000][12649][gc ] Periodic GC disabled
    [2020-09-01T19:04:47.263+0000][12649][gc,heap,exit ] Heap
    [2020-09-01T19:04:47.263+0000][12649][gc,heap,exit ] garbage-first heap total 1048576K, used 1024K [0x00000000c
    0000000, 0x0000000100000000)
    [2020-09-01T19:04:47.263+0000][12649][gc,heap,exit ] region size 1024K, 2 young (2048K), 0 survivors (0K)
    [2020-09-01T19:04:47.263+0000][12649][gc,heap,exit ] Metaspace used 3123K, capacity 4480K, committed 4480K,
    reserved 1056768K
    [2020-09-01T19:04:47.263+0000][12649][gc,heap,exit ] class space used 263K, capacity 384K, committed 384K, re
    served 1048576K
    [root@ElasticEDT elasticsearch]#

I hope this helps, to clarify what the problem is, and why Elasticsearch analytics engine does not start for us!
Greetings from Bucharest, Bogdan

6

Can you try runnung elasticsearch manually to see what happens ?
Use this : /usr/share/elasticsearch/bin/elasticsearch

When your install RPM package, ES is installed in :

  • /etc/elasticsearch (config file),
  • /usr/share/elasticsearch (binaries)
1 Like
7

hi Yassine,
thanks for your reply earlier, and your explanation that ES is installed in:

  • /etc/elasticsearch (config file),
  • /usr/share/elasticsearch (binaries)
    I checked for myself and I'm happy to confirm this is so, the installation makes more sense to me now!
8

here is the output when trying to run Elasticsearch manually, as you suggested:
[root@ElasticEDT bin]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@ElasticEDT bin]# pwd
/usr/share/elasticsearch/bin
[root@ElasticEDT bin]# ./elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-09-02T18:34:28,284][WARN ][o.e.c.l.LogConfigurator ] [ElasticEDT] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace %node_name with [%node_name]%marker in these locations:
/etc/elasticsearch/test/log4j2.properties
[2020-09-02T18:34:28,367][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ElasticEDT] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.1.jar:7.8.1]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
2020-09-02 15:34:28,717860 UTC [25395] ERROR CLogger.cc@311 Cannot log to named pipe /tmp/elasticsearch-10919022139451207318/controller_log_25222 as it could not be opened for writing
2020-09-02 15:34:28,717972 UTC [25395] INFO Main.cc@104 Parent process died - ML controller exiting
[root@ElasticEDT bin]#

9

here is the error

java.lang.RuntimeException: can not run elasticsearch as root

10

so it looks like I cannot run Elasticsearch as root:
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
my question is: what user should I run it under, and why is this not stated in your online documentation below:
https://www.elastic.co/guide/en/elastic-stack-get-started/7.8/get-started-elastic-stack.html

11

hi Yassine,
thanks for your update, we both replied at the same time: please see my previous post below: which user should it run under?
thanks, Bogdan

12

logout from root and use any normal user with sudo

13

hi Yassine,
thanks for this, I did this and set the new password for the elasticsearch user.
Now I am getting this error below:
bash-4.4$ ./elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020-09-02 19:37:53,812 main ERROR Unable to access file:///etc/elasticsearch/test/log4j2.properties java.io.FileNotFoundException: /etc/elasticsearch/test/log4j2.properties (Permission denied)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184)
at java.base/java.net.URL.openStream(URL.java:1140)
at org.apache.logging.log4j.core.config.ConfigurationSource.fromUri(ConfigurationSource.java:247)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:237)
at org.elasticsearch.common.logging.LogConfigurator$3.visitFile(LogConfigurator.java:224)
at org.elasticsearch.common.logging.LogConfigurator$3.visitFile(LogConfigurator.java:220)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2724)
at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:220)
at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:129)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:354)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

Exception in thread "main" java.lang.NullPointerException
at org.apache.logging.log4j.core.config.composite.CompositeConfiguration.(CompositeConfiguration.java:65)
at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:236)
at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:129)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:354)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
bash-4.4$

14

hi Yassine,
I have fixed the obvious file and directory permissions errors above, but now I am getting another error, see below for more info:
bash-4.4$ ./elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-09-02T19:45:14,029][WARN ][o.e.c.l.LogConfigurator ] [ElasticEDT] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace %node_name with [%node_name]%marker in these locations:
/etc/elasticsearch/test/log4j2.properties
[2020-09-02T19:45:14,270][INFO ][o.e.n.Node ] [ElasticEDT] version[7.8.1], pid[27288], build[default/rpm/b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89/2020-07-21T16:40:44.668009Z], OS[Linux/4.18.0-147.5.1.el8_1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.8/11.0.8+10-LTS]
[2020-09-02T19:45:14,271][INFO ][o.e.n.Node ] [ElasticEDT] JVM home [/usr/lib/jvm/java-11-openjdk-11.0.8.10-0.el8_2.x86_64]
[2020-09-02T19:45:14,272][INFO ][o.e.n.Node ] [ElasticEDT] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-17844278366693308801, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-09-02T19:45:16,175][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [aggs-matrix-stats]
[2020-09-02T19:45:16,176][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [analysis-common]
[2020-09-02T19:45:16,176][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [constant-keyword]
[2020-09-02T19:45:16,177][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [flattened]
... and so on ...
[transform]
[2020-09-02T19:45:16,182][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [transport-netty4]
[2020-09-02T19:45:16,182][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [vectors]
[2020-09-02T19:45:16,183][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-analytics]
[2020-09-02T19:45:16,183][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-async-search]
[2020-09-02T19:45:16,183][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-autoscaling]
[2020-09-02T19:45:16,183][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-ccr]
[2020-09-02T19:45:16,184][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-core]
[2020-09-02T19:45:16,184][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-deprecation]
[2020-09-02T19:45:16,184][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-enrich]
[2020-09-02T19:45:16,184][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-eql]
[2020-09-02T19:45:16,184][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-graph]
[2020-09-02T19:45:16,185][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-identity-provider]
[2020-09-02T19:45:16,185][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-ilm]
[2020-09-02T19:45:16,185][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-logstash]
[2020-09-02T19:45:16,185][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-ml]
[2020-09-02T19:45:16,186][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-monitoring]
[2020-09-02T19:45:16,186][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-ql]
[2020-09-02T19:45:16,186][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-rollup]
[2020-09-02T19:45:16,186][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-security]
[2020-09-02T19:45:16,187][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-sql]
[2020-09-02T19:45:16,187][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-voting-only-node]
[2020-09-02T19:45:16,187][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-watcher]
[2020-09-02T19:45:16,187][INFO ][o.e.p.PluginsService ] [ElasticEDT] no plugins loaded
[2020-09-02T19:45:16,234][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ElasticEDT] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.1.jar:7.8.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:301) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:223) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:216) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
Likely root cause: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:216)
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271)
at org.elasticsearch.node.Node.(Node.java:335)
at org.elasticsearch.node.Node.(Node.java:266)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
bash-4.4$

15

You still have permissions issue to /var/lib/elasticsearch/nodes/

16

hi Yassine,
I have opened this nodes folder in the maximal possible way, 777:
[root@ElasticEDT elasticsearch]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@ElasticEDT elasticsearch]# pwd
/var/lib/elasticsearch
[root@ElasticEDT elasticsearch]# ll
total 0
drwxrwsrwx. 3 elasticsearch elasticsearch 15 Mar 30 22:28 nodes
[root@ElasticEDT elasticsearch]#
and I also changed the owner to elasticsearch, as you can see, and yet the problem is still happening, see below:
,,, and so on ...
[2020-09-02T20:02:21,218][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-monitoring]
[2020-09-02T20:02:21,219][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-ql]
[2020-09-02T20:02:21,219][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-rollup]
[2020-09-02T20:02:21,219][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-security]
[2020-09-02T20:02:21,219][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-sql]
[2020-09-02T20:02:21,220][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-voting-only-node]
[2020-09-02T20:02:21,220][INFO ][o.e.p.PluginsService ] [ElasticEDT] loaded module [x-pack-watcher]
[2020-09-02T20:02:21,220][INFO ][o.e.p.PluginsService ] [ElasticEDT] no plugins loaded
[2020-09-02T20:02:21,266][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ElasticEDT] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.1.jar:7.8.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:301) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:223) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - 2020-04-08 08:55:42]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:216) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:335) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.node.Node.(Node.java:266) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.1.jar:7.8.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.1.jar:7.8.1]
... 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
Likely root cause: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125)
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:216)
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271)
at org.elasticsearch.node.Node.(Node.java:335)
at org.elasticsearch.node.Node.(Node.java:266)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
bash-4.4$

17

The /var/lib/elasticsearch/nodes/0/node.lock is from the old installation
is there any data in your node from the old installation in /var/lib/elasticsearch/nodes/0 ?

18

No, there is no data below from the old installation:
[root@ElasticEDT 0]# pwd
/var/lib/elasticsearch/nodes/0
[root@ElasticEDT 0]# ll
total 0
-rwxr-xr-x. 1 root root 0 Mar 30 22:28 node.lock
drwxr-sr-x. 2 root root 121 Mar 31 13:47 _state
[root@ElasticEDT 0]#

should I delete this folder?

19

remove /var/lib/elasticsearch/nodes/0/node.lock

20

I have removed this node.lock, see below:
[root@ElasticEDT 0]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@ElasticEDT 0]# pwd
/var/lib/elasticsearch/nodes/0
[root@ElasticEDT 0]# ll
total 0
drwxr-sr-x. 2 root root 121 Mar 31 13:47 _state
[root@ElasticEDT 0]#