Elasticsearch Service won't start

Elastic Stack Elasticsearch
1

After adding some memory to our server and rebooting it afterward, Elasticsearch does not start anymore.

I am using ES 7.8.0

I execute sudo systemctl start elasticsearch.service

and get the following error

Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Executing sudo systemctl status elasticsearch.service gives

Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]: Error: Could not create the Java Virtual Machine.
Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]: Error: A fatal exception has occurred. Program will exit.
Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]:         at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.ja
Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]:         at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomi
Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]:         at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:5
Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsPar
Jul 23 00:17:00 SERVER1 systemd-entrypoint[18020]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.ja
Jul 23 00:17:00 SERVER1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 23 00:17:00 SERVER1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jul 23 00:17:00 SERVER1 systemd[1]: Failed to start Elasticsearch.

The log file just says

[2020-07-22T10:24:36,607][INFO ][o.e.n.Node               ] [SERVER1] version[7.8.0], pid[13393], build[default/deb/757314695644ea9a1dc2$
[2020-07-22T10:24:36,665][INFO ][o.e.n.Node               ] [SERVER1] JVM home [/usr/share/elasticsearch/jdk]
[2020-07-22T10:24:36,666][INFO ][o.e.n.Node               ] [SERVER1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -De$

Any suggestions?

1 Like
2

Which JVM?
Did you change the heap settings as well, and if so what do they look like?
Can you try running Elasticsearch directly and seeing what it says?

3

I actually don't know which JVM. I am also confused why it says JVM. I didn't change any other settings.

Running Elasticsearch directly with sudo says the following:

future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
[2020-07-23T00:26:08,270][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SERVER1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
        ... 6 more
uncaught exception in thread [main]
java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
2020-07-22 22:26:09,029755 UTC [19466] ERROR CNamedPipeFactory.cc@226 Unable to open named pipe /tmp/elasticsearch-8865269252892219593/controller_log_19288 for writing: Interrupted system call
2020-07-22 22:26:09,029847 UTC [19466] ERROR CLogger.cc@298 Cannot log to named pipe /tmp/elasticsearch-8865269252892219593/controller_log_19288 as it could not be opened for writing
2020-07-22 22:26:09,029867 UTC [19466] FATAL Main.cc@102 Could not reconfigure logging

Without sudo it says

future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
./bin/elasticsearch-env: line 81: /etc/default/elasticsearch: Permission denied
4

What does java --version show?

5

java -version

java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

and java --version

Unrecognized option: --version
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
6

Ahh sorry, had an extra -! Thanks for that.

It seems odd that it'd be erroring like that without more in the logs. Are you sure there's nothing else there? Can you try restarting the service and tailing the log file at the same time, to see what happens? Is there anything in the OS level logs that might be related?

7

The log file /var/log/elasticsearch/elasticsearch.log says

java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.8.0.jar:7.8.0]
[2020-07-23T00:26:08,270][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [df-aiprod001] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
        ... 6 more

But I guess that is from the direct start using sudo. Syslog outputs the following

Jul 23 00:37:37 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:37Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:37 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:37Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:40 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:40Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:40 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:40Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:42 SERVER1 systemd[1]: Starting Elasticsearch...
Jul 23 00:37:42 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:42Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:42 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:42Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: output:
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: [0.001s][error][logging] Error opening log file '/var/log/elasticsearch/gc.log': Permission denied
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: [0.001s][error][logging] Initialization of output 'file=/var/log/elasticsearch/gc.log' using options 'filecount=32,filesize=64m' failed.
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: error:
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: OpenJDK 64-Bit Server VM warning: Ignoring option UseConcMarkSweepGC; support was removed in 14.0
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: OpenJDK 64-Bit Server VM warning: Ignoring option CMSInitiatingOccupancyFraction; support was removed in 14.0
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: OpenJDK 64-Bit Server VM warning: Ignoring option UseCMSInitiatingOccupancyOnly; support was removed in 14.0
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: Error: Could not create the Java Virtual Machine.
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: Error: A fatal exception has occurred. Program will exit.
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: #011at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: #011at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: #011at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: #011at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
Jul 23 00:37:43 SERVER1 systemd-entrypoint[21213]: #011at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
Jul 23 00:37:43 SERVER1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 23 00:37:43 SERVER1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jul 23 00:37:43 SERVER1 systemd[1]: Failed to start Elasticsearch.
Jul 23 00:37:45 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:45Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:45 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:45Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:47 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:47Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:47 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:47Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:50 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:50Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:50 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:50Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","elasticsearch","monitoring"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","elasticsearch","monitoring"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","plugins","licensing"],"pid":1139,"message":"License information could not be obtained from Elasticsearch due to Error: No Li$
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","elasticsearch","data"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","elasticsearch","data"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","plugins","licensing"],"pid":1139,"message":"License information could not be obtained from Elasticsearch due to Error: No Li$
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:52 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:52Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:55 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:55Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:55 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:55Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:37:57 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:57Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:37:57 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:37:57Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:38:00 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:00Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:38:00 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:00Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:38:02 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:02Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:38:02 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:02Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:38:05 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:05Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:38:05 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:05Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:38:07 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:07Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:38:07 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:07Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:38:10 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:10Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 23 00:38:10 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:10Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"No living connections"}
Jul 23 00:38:12 SERVER1 kibana[1139]: {"type":"log","@timestamp":"2020-07-22T22:38:12Z","tags":["warning","elasticsearch","admin"],"pid":1139,"message":"Unable to revive connection: http://localhost:9200/"}

It looks like the kibana error occurs every fews seconds all day long.

8

Is there anything useful in there, or any other files in that directory?

9 
[2020-07-22T08:24:32.052+0000][13393][gc,heap] Heap region size: 4M
[2020-07-22T08:24:32.052+0000][13393][gc,heap,coops] Heap address: 0x00000004c0000000, size: 13312 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
[2020-07-22T08:24:34.646+0000][13393][gc           ] Using G1
[2020-07-22T08:24:34.647+0000][13393][gc,cds       ] Mark closed archive regions in map: [0x00000007ffc00000, 0x00000007ffc7eff8]
[2020-07-22T08:24:34.647+0000][13393][gc,cds       ] Mark open archive regions in map: [0x00000007ffb00000, 0x00000007ffb51ff8]
[2020-07-22T08:24:34.662+0000][13393][gc           ] Periodic GC disabled
[2020-07-22T08:24:35.312+0000][13393][safepoint    ] Safepoint "ICBufferFull", Time since last: 655197463 ns, Reaching safepoint: 216640 ns, At safepoint: 14059 ns, Total: 230699 ns
[2020-07-22T08:24:36.059+0000][13393][safepoint    ] Safepoint "ICBufferFull", Time since last: 746514458 ns, Reaching safepoint: 193166 ns, At safepoint: 9172 ns, Total: 202338 ns
[2020-07-22T08:24:36.660+0000][13393][safepoint    ] Safepoint "ICBufferFull", Time since last: 601130252 ns, Reaching safepoint: 208655 ns, At safepoint: 5710 ns, Total: 214365 ns
[2020-07-22T08:24:37.925+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1264990584 ns, Reaching safepoint: 204693 ns, At safepoint: 3716 ns, Total: 208409 ns
[2020-07-22T08:24:38.926+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1000165747 ns, Reaching safepoint: 106172 ns, At safepoint: 3114 ns, Total: 109286 ns
[2020-07-22T08:24:47.927+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 9000858776 ns, Reaching safepoint: 225371 ns, At safepoint: 3288 ns, Total: 228659 ns
[2020-07-22T08:24:48.927+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1000160188 ns, Reaching safepoint: 104216 ns, At safepoint: 3171 ns, Total: 107387 ns
[2020-07-22T08:24:54.928+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 6000605511 ns, Reaching safepoint: 106664 ns, At safepoint: 4197 ns, Total: 110861 ns
[2020-07-22T08:25:01.929+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 7000642274 ns, Reaching safepoint: 136394 ns, At safepoint: 3870 ns, Total: 140264 ns
[2020-07-22T08:25:02.929+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1000146663 ns, Reaching safepoint: 205359 ns, At safepoint: 4375 ns, Total: 209734 ns
[2020-07-22T08:25:07.920+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 4991140523 ns, Reaching safepoint: 162170 ns, At safepoint: 9409 ns, Total: 171579 ns
[2020-07-22T08:25:08.921+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1000330417 ns, Reaching safepoint: 102010 ns, At safepoint: 3142 ns, Total: 105152 ns
[2020-07-22T08:25:10.232+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1311118463 ns, Reaching safepoint: 169313 ns, At safepoint: 2693 ns, Total: 172006 ns
[2020-07-22T08:25:12.232+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 2000239571 ns, Reaching safepoint: 118087 ns, At safepoint: 3300 ns, Total: 121387 ns
[2020-07-22T08:25:22.233+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 10000978830 ns, Reaching safepoint: 128259 ns, At safepoint: 3295 ns, Total: 131554 ns
[2020-07-22T08:25:23.234+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1000142816 ns, Reaching safepoint: 121376 ns, At safepoint: 3095 ns, Total: 124471 ns
[2020-07-22T08:25:23.342+0000][13393][gc,start     ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2020-07-22T08:25:23.342+0000][13393][gc,task      ] GC(0) Using 4 workers of 4 for evacuation
[2020-07-22T08:25:23.342+0000][13393][gc,age       ] GC(0) Desired survivor size 44040192 bytes, new threshold 15 (max threshold 15)
[2020-07-22T08:25:23.352+0000][13393][gc,age       ] GC(0) Age table with threshold 15 (max threshold 15)
[2020-07-22T08:25:23.352+0000][13393][gc,age       ] GC(0) - age   1:   15539960 bytes,   15539960 total
[2020-07-22T08:25:23.352+0000][13393][gc,phases    ] GC(0)   Pre Evacuate Collection Set: 0.3ms
[2020-07-22T08:25:23.352+0000][13393][gc,phases    ] GC(0)   Merge Heap Roots: 0.1ms
[2020-07-22T08:25:23.352+0000][13393][gc,phases    ] GC(0)   Evacuate Collection Set: 8.8ms
[2020-07-22T08:25:23.352+0000][13393][gc,phases    ] GC(0)   Post Evacuate Collection Set: 1.3ms
[2020-07-22T08:25:23.352+0000][13393][gc,phases    ] GC(0)   Other: 0.2ms
[2020-07-22T08:25:23.352+0000][13393][gc,heap      ] GC(0) Eden regions: 166->0(162)
[2020-07-22T08:25:23.352+0000][13393][gc,heap      ] GC(0) Survivor regions: 0->4(21)
[2020-07-22T08:25:23.352+0000][13393][gc,heap      ] GC(0) Old regions: 0->0
[2020-07-22T08:25:23.352+0000][13393][gc,heap      ] GC(0) Archive regions: 2->2
[2020-07-22T08:25:23.352+0000][13393][gc,heap      ] GC(0) Humongous regions: 0->0
[2020-07-22T08:25:23.352+0000][13393][gc,metaspace ] GC(0) Metaspace: 19006K(20096K)->19006K(20096K) NonClass: 16654K(17408K)->16654K(17408K) Class: 2351K(2688K)->2351K(2688K)
[2020-07-22T08:25:23.352+0000][13393][gc           ] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 667M->18M(13312M) 10.818ms
[2020-07-22T08:25:23.352+0000][13393][gc,cpu       ] GC(0) User=0.04s Sys=0.01s Real=0.01s
[2020-07-22T08:25:23.352+0000][13393][safepoint    ] Safepoint "G1CollectForAllocation", Time since last: 107737877 ns, Reaching safepoint: 84491 ns, At safepoint: 10899291 ns, Total: 10983782 ns
[2020-07-22T08:25:23.354+0000][13393][safepoint    ] Safepoint "BulkRevokeBias", Time since last: 1583980 ns, Reaching safepoint: 129262 ns, At safepoint: 64772 ns, Total: 194034 ns
[2020-07-22T08:25:23.354+0000][13393][safepoint    ] Safepoint "BulkRevokeBias", Time since last: 167970 ns, Reaching safepoint: 117506 ns, At safepoint: 44150 ns, Total: 161656 ns
[2020-07-22T08:25:23.357+0000][13393][safepoint    ] Safepoint "BulkRevokeBias", Time since last: 2071916 ns, Reaching safepoint: 42437 ns, At safepoint: 46050 ns, Total: 88487 ns
[2020-07-22T08:25:23.357+0000][13393][safepoint    ] Safepoint "BulkRevokeBias", Time since last: 52426 ns, Reaching safepoint: 109908 ns, At safepoint: 46244 ns, Total: 156152 ns
[2020-07-22T08:25:24.922+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1565354034 ns, Reaching safepoint: 131995 ns, At safepoint: 4295 ns, Total: 136290 ns
[2020-07-22T08:25:26.923+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 2000242062 ns, Reaching safepoint: 151481 ns, At safepoint: 5343 ns, Total: 156824 ns
[2020-07-22T08:25:28.532+0000][13393][gc,start     ] GC(1) Pause Young (Concurrent Start) (Metadata GC Threshold)
[2020-07-22T08:25:28.532+0000][13393][gc,task      ] GC(1) Using 4 workers of 4 for evacuation
[2020-07-22T08:25:28.532+0000][13393][gc,age       ] GC(1) Desired survivor size 44040192 bytes, new threshold 15 (max threshold 15)
[2020-07-22T08:25:28.541+0000][13393][gc,age       ] GC(1) Age table with threshold 15 (max threshold 15)
[2020-07-22T08:25:28.541+0000][13393][gc,age       ] GC(1) - age   1:    2218088 bytes,    2218088 total
[2020-07-22T08:25:28.541+0000][13393][gc,age       ] GC(1) - age   2:   14935184 bytes,   17153272 total
[2020-07-22T08:25:28.541+0000][13393][gc,phases    ] GC(1)   Pre Evacuate Collection Set: 0.4ms
[2020-07-22T08:25:28.541+0000][13393][gc,phases    ] GC(1)   Merge Heap Roots: 0.1ms
[2020-07-22T08:25:28.541+0000][13393][gc,phases    ] GC(1)   Evacuate Collection Set: 7.3ms
[2020-07-22T08:25:28.541+0000][13393][gc,phases    ] GC(1)   Post Evacuate Collection Set: 0.9ms
[2020-07-22T08:25:28.541+0000][13393][gc,phases    ] GC(1)   Other: 0.5ms
[2020-07-22T08:25:28.541+0000][13393][gc,heap      ] GC(1) Eden regions: 46->0(161)
[2020-07-22T08:25:28.541+0000][13393][gc,heap      ] GC(1) Survivor regions: 4->5(21)
[2020-07-22T08:25:28.541+0000][13393][gc,heap      ] GC(1) Old regions: 0->0
[2020-07-22T08:25:28.541+0000][13393][gc,heap      ] GC(1) Archive regions: 2->2
[2020-07-22T08:25:28.541+0000][13393][gc,heap      ] GC(1) Humongous regions: 0->0
[2020-07-22T08:25:28.541+0000][13393][gc,metaspace ] GC(1) Metaspace: 20043K(21248K)->20043K(21248K) NonClass: 17590K(18432K)->17590K(18432K) Class: 2452K(2816K)->2452K(2816K)
[2020-07-22T08:25:28.541+0000][13393][gc           ] GC(1) Pause Young (Concurrent Start) (Metadata GC Threshold) 202M->21M(13312M) 9.286ms
[2020-07-22T08:25:28.541+0000][13393][gc,cpu       ] GC(1) User=0.03s Sys=0.00s Real=0.01s
[2020-07-22T08:25:28.541+0000][13393][gc           ] GC(2) Concurrent Cycle
[2020-07-22T08:25:28.541+0000][13393][gc,marking   ] GC(2) Concurrent Clear Claimed Marks
[2020-07-22T08:25:28.541+0000][13393][gc,marking   ] GC(2) Concurrent Clear Claimed Marks 0.042ms
[2020-07-22T08:25:28.541+0000][13393][gc,marking   ] GC(2) Concurrent Scan Root Regions
[2020-07-22T08:25:28.541+0000][13393][safepoint    ] Safepoint "CollectForMetadataAllocation", Time since last: 1608902768 ns, Reaching safepoint: 65729 ns, At safepoint: 9569222 ns, Total: 9634951 ns
[2020-07-22T08:25:28.544+0000][13393][safepoint    ] Safepoint "BulkRevokeBias", Time since last: 2327419 ns, Reaching safepoint: 137364 ns, At safepoint: 67559 ns, Total: 204923 ns
[2020-07-22T08:25:28.544+0000][13393][safepoint    ] Safepoint "BulkRevokeBias", Time since last: 32772 ns, Reaching safepoint: 119619 ns, At safepoint: 67460 ns, Total: 187079 ns
[2020-07-22T08:25:28.547+0000][13393][gc,marking   ] GC(2) Concurrent Scan Root Regions 5.917ms
[2020-07-22T08:25:28.547+0000][13393][gc,marking   ] GC(2) Concurrent Mark (56.568s)
[2020-07-22T08:25:28.547+0000][13393][gc,marking   ] GC(2) Concurrent Mark From Roots
[2020-07-22T08:25:28.547+0000][13393][gc,task      ] GC(2) Using 1 workers of 1 for marking
[2020-07-22T08:25:28.552+0000][13393][gc,marking   ] GC(2) Concurrent Mark From Roots 4.654ms
[2020-07-22T08:25:28.552+0000][13393][gc,marking   ] GC(2) Concurrent Preclean
[2020-07-22T08:25:28.552+0000][13393][gc,marking   ] GC(2) Concurrent Preclean 0.065ms
[2020-07-22T08:25:28.552+0000][13393][gc,marking   ] GC(2) Concurrent Mark (56.568s, 56.573s) 4.766ms
[2020-07-22T08:25:28.552+0000][13393][gc,start     ] GC(2) Pause Remark
[2020-07-22T08:25:28.555+0000][13393][gc           ] GC(2) Pause Remark 25M->25M(13312M) 2.727ms
[2020-07-22T08:25:28.555+0000][13393][gc,cpu       ] GC(2) User=0.01s Sys=0.00s Real=0.01s
[2020-07-22T08:25:28.555+0000][13393][safepoint    ] Safepoint "G1Concurrent", Time since last: 7883780 ns, Reaching safepoint: 148760 ns, At safepoint: 2779087 ns, Total: 2927847 ns
[2020-07-22T08:25:28.555+0000][13393][gc,marking   ] GC(2) Concurrent Rebuild Remembered Sets
[2020-07-22T08:25:28.555+0000][13393][gc,marking   ] GC(2) Concurrent Rebuild Remembered Sets 0.595ms
[2020-07-22T08:25:28.556+0000][13393][gc,start     ] GC(2) Pause Cleanup
[2020-07-22T08:25:28.556+0000][13393][gc           ] GC(2) Pause Cleanup 27M->27M(13312M) 0.317ms
[2020-07-22T08:25:28.556+0000][13393][gc,cpu       ] GC(2) User=0.00s Sys=0.00s Real=0.00s
[2020-07-22T08:25:28.556+0000][13393][safepoint    ] Safepoint "G1Concurrent", Time since last: 651898 ns, Reaching safepoint: 479302 ns, At safepoint: 352200 ns, Total: 831502 ns
[2020-07-22T08:25:28.556+0000][13393][gc,marking   ] GC(2) Concurrent Cleanup for Next Mark
[2020-07-22T08:25:28.596+0000][13393][gc,marking   ] GC(2) Concurrent Cleanup for Next Mark 39.670ms
[2020-07-22T08:25:28.596+0000][13393][gc           ] GC(2) Concurrent Cycle 55.002ms
[2020-07-22T08:25:29.557+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 1000092721 ns, Reaching safepoint: 121855 ns, At safepoint: 3345 ns, Total: 125200 ns
[2020-07-22T08:25:31.557+0000][13393][safepoint    ] Safepoint "Cleanup", Time since last: 2000239350 ns, Reaching safepoint: 98851 ns, At safepoint: 3280 ns, Total: 102131 ns
[2020-07-22T08:25:32.340+0000][13393][gc,heap,exit ] Heap
[2020-07-22T08:25:32.340+0000][13393][gc,heap,exit ]  garbage-first heap   total 13631488K, used 332953K [0x00000004c0000000, 0x0000000800000000)
[2020-07-22T08:25:32.340+0000][13393][gc,heap,exit ]   region size 4096K, 82 young (335872K), 5 survivors (20480K)
[2020-07-22T08:25:32.340+0000][13393][gc,heap,exit ]  Metaspace       used 20574K, capacity 22145K, committed 22396K, reserved 1069056K
[2020-07-22T08:25:32.340+0000][13393][gc,heap,exit ]   class space    used 2550K, capacity 2923K, committed 2940K, reserved 1048576K

The problem is, I am not an expert and don't know which files are useful.

10

That's ok, I haven't seen this either so it just a matter of poking around :slight_smile:

I can't see anything there. I'll see if someone else might have an idea.

11

Did you update the server, too, like yum/apt-get update before reboot? I suspect the reboot is part of the cause, changing paths or undoing anything special that may have been done long ago to get things running, like JVM selection, etc.

How many java JVMs on this server, and are they conflicting? I believe V7 has its own JVM so you might make sure the OS one is removed, i.e. I'd not expect java -version to work if ES is using an embedded Java.

Part of your test shows this JVM when running directly: /opt/jdk/jdk1.8.0_202/jre which is old and not what you want BUT the error there is about the ROOT user, so just try that again with the elasticsearch user and see how it works.

From log looks like right JVM in /usr/share/elasticsearch/jdk - can you go there and test that Java?

I ran some tests, using CentOS (not Debian), which has native Java installed. In my test 7.8 bundles OpenJVM version 14 and log is similar to yours but after the long JVM arguments line you get a series of 'loaded module' lines then data path, heap size, etc.

Also odd that it seems either Kibana or ES is starting with options removed in the embedded JDK versions like UseConcMarkSweepGC - that is weird.

You also now have permission problems in /var/log/elasticsearch/ because you did a root start and now the elasticsearch user can't write to them - so need do a "chown -R elasticsearch /var/log/elasticsearch"

So fix that and the JVM should start.

Finally, when you post your logs, they are getting cut off so can't see the ends of the lines, which makes it hard.

12

@Jensxy

You are using JDK8 but -Xlog:gc* is supported on JDK9+. Can you check / post file /etc/elasticsearch/jvm.options?

In the startup logs (elasticsearch.log) there will be two log entries. Search for " JVM home" and "JVM arguments".

Run <JVM.home Value>/bin/java -version and post output

13

@Vinayak_Sapre

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms13g
-Xmx13g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
# 10-:-XX:+UseG1GC
# 10-:-XX:G1ReservePercent=25
# 10-:-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

And ./bin/java -version returns

java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode
14

@Steve_Mushero thank you for your reply. Unfortunately, restoring the configuration at a time before the reboot does not help. Executing ps -ef | grep java returns 14795 12566 0 09:45 pts/0 00:00:00 grep --color=auto java.

How do I start it as the elasticsearch user? What command do I have do use?

I did chown -R elasticsearch /var/log/elasticsearch but nothing changed.

Executing ./bin/java -versionin /usr/share/elasticsearch/jdkreturns

openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)
15

Well, our goal is start via normal scripts if we can - the ownership should have fixed the permission error in the logs when you did that.

Use 'sudo -u elasticsearch {start command line} ' to start as other user directly if you want; not as root.

16

Okay, sudo -u elasticsearch ./bin/elasticsearch returns

future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet thi                                                                                                                  s requirement
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet thi                                                                                                                  s requirement
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /var/log/elasticsearch/gc.log due to Permission denied

2020-07-23 10:06:31,114 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_server.json java.io.IOExcepti                                                                                                                  on: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createNewFile(File.java:1012)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(Rollin                                                                                                                  gFileManager.java:628)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(Rollin                                                                                                                  gFileManager.java:608)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
        at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
        at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
        at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:236)
        at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:129)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:354)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

2020-07-23 10:06:31,118 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFile                                                                                                                  Appender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.r                                                                                                                  olling.RollingFileManager$RollingFileManagerFactory@5386659f] unable to create manager for [/var/log/elasticsearch/elastics                                                                                                                  earch_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@14028087[patter                                                                                                                  n=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=Com                                                                                                                  positeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTrigg                                                                                                                  eringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseUR                                                                                                                  I=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p",                                                                                                                   "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%en                                                                                                                  c{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.ap                                                                                                                  pender.rolling.RollingFileManager$RollingFileManagerFactory@5386659f] unable to create manager for [/var/log/elasticsearch/                                                                                                                  elasticsearch_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@1402808                                                                                                                  7[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, po                                                                                                                  licy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBa                                                                                                                  sedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), adv                                                                                                                  ertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level                                                                                                                  ": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notE                                                                                                                  mpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]]
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
        at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
        at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
        at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:236)
        at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:129)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:354)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

2020-07-23 10:06:31,122 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch_deprecation.log java.io.IOExc     
...

The output is actually longer than I can post. I've also added the output of java in /usr/share/elasticsearch/jdkin my last post.

17

You have to run that chown again as it can't create the file - we have to get beyond that so elasticsearch user can create the files or it'll never start.

18

I first executedsudo chown -R elasticsearch /var/log/elasticsearch and then I executed sudo -u elasticsearch ./bin/elasticsearch, however, the same error messages occur.

The permissions are the following: d-----S--- 2 elasticsearch elasticsearch 36864 Jul 23 09:39 elasticsearch

19

Ah, something seriously wrong with that directory - what in the world happened? This makes me worry other things are messed up - I assume this is a good VM, not Docker or on NFS or something weird (and not K8S).

So have to fix that; should look like this:
ls -ld /var/log/elasticsearch/
drwxr-s--- 2 elasticsearch elasticsearch 77824 Jul 23 07:38 elasticsearch

So, do a chmod 750 /var/log/elasticsearch/ to fix that, and erase everything in it so you have a clean start:

rm -rf /var/log/elasticsearch/*

Then start with the normal init script

20

Okay, at least now there's a new error. The permissions are now drwxr-s--- 2 elasticsearch elasticsearch 36864 Jul 23 11:12 /var/log/elasticsearch/.

The output of sudo -u elasticsearch ./bin/elasticsearchshows

future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
2020-07-23 11:13:54,917 main ERROR RollingFileManager (/var/log/elasticsearch/elasticsearch_deprecation.log) java.io.FileNotFoundException: /var/log/elasticsearch/elasticsearch_deprecation.log (Permission denied) java.io.FileNotFoundException: /var/log/elasticsearch/elasticsearch_deprecation.log (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
		...

2020-07-23 11:13:54,921 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@451001e5] unable to create manager for [/var/log/elasticsearch/elasticsearch_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@2b40ff9c[pattern=/var/log/elasticsearch/elasticsearch_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@451001e5] unable to create manager for [/var/log/elasticsearch/elasticsearch_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@2b40ff9c[pattern=/var/log/elasticsearch/elasticsearch_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n, filePermissions=null, fileOwner=null]]
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
		...
...

2020-07-23 11:13:55,009 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@451001e5] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@125290e5[pattern=/var/log/elasticsearch/elasticsearch_index_indexing_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "index_indexing_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%ESMessageField{message}"}%notEmpty{, "took": "%ESMessageField{took}"}%notEmpty{, "took_millis": "%ESMessageField{took_millis}"}%notEmpty{, "doc_type": "%ESMessageField{doc_type}"}%notEmpty{, "id": "%ESMessageField{id}"}%notEmpty{, "routing": "%ESMessageField{routing}"}%notEmpty{, "source": "%ESMessageField{source}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@451001e5] unable to create manager for [/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@125290e5[pattern=/var/log/elasticsearch/elasticsearch_index_indexing_slowlog-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "index_indexing_slowlog", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name"%notEmpty{, "message": "%ESMessageField{message}"}%notEmpty{, "took": "%ESMessageField{took}"}%notEmpty{, "took_millis": "%ESMessageField{took_millis}"}%notEmpty{, "doc_type": "%ESMessageField{doc_type}"}%notEmpty{, "id": "%ESMessageField{id}"}%notEmpty{, "routing": "%ESMessageField{routing}"}%notEmpty{, "source": "%ESMessageField{source}"}%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
}, filePermissions=null, fileOwner=null]]
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
        at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
		...

2020-07-23 11:13:55,010 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
        ...

2020-07-23 11:13:55,010 main ERROR Null object returned for RollingFile in Appenders.
2020-07-23 11:13:55,010 main ERROR Null object returned for RollingFile in Appenders.
2020-07-23 11:13:55,012 main ERROR Unable to locate appender "index_indexing_slowlog_rolling_old" for logger config "index.indexing.slowlog.index"
2020-07-23 11:13:55,012 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
2020-07-23 11:13:55,012 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail"
2020-07-23 11:13:55,012 main ERROR Unable to locate appender "index_search_slowlog_rolling_old" for logger config "index.search.slowlog"
2020-07-23 11:13:55,012 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
2020-07-23 11:13:55,013 main ERROR Unable to locate appender "deprecation_rolling_old" for logger config "org.elasticsearch.deprecation"
2020-07-23 11:13:55,013 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
[2020-07-23T11:13:55,219][INFO ][o.e.n.Node               ] [SERVER1] version[7.8.0], pid[26529], build[default/deb/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/4.15.0-108-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_202/25.202-b08]
[2020-07-23T11:13:55,222][INFO ][o.e.n.Node               ] [SERVER1] JVM home [/opt/jdk/jdk1.8.0_202/jre]
[2020-07-23T11:13:55,222][INFO ][o.e.n.Node               ] [SERVER1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,JRE, -Xms13g, -Xmx13g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-9054766181985020957, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=6979321856, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-07-23T11:13:58,154][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SERVER1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.5.0 but version 7.8.0 is running
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.5.0 but version 7.8.0 is running
        at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:348) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:533) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]