Elasticsearch wont run

Elastic Stack Elasticsearch
1

Hi dears
I have upgraded the Elasticsearch and removed old version and copy all the new Elasticsearch parts in the same directory
(upgrading ES 7.6.2 to 7.15.2)
Unfortunately when I want to start the service with sysV this exception occurs:

[root@test-elk2 tools]# service elasticsearch start
Starting elasticsearch (via systemctl):  Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
                                                           [FAILED]
[root@test-elk2 tools]# journalctl -xe
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: Non-option arguments:
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: command
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: Option             Description
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: ------             -----------
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -E <KeyValuePair>  Configure a setting
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -h, --help         Show help
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -s, --silent       Show minimal output
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -v, --verbose      Show verbose output
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: ERROR: Unknown command [has-passwd]
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: Exception in thread "main" java.lang.IllegalArgumentException: expected one argument specifying path to jvm.options but was [/etc/elasticsearch, /usr/share/elasticsearch
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:59)
Nov 29 10:05:09 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 10:05:09 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 10:05:09 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 10:05:09 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 10:05:09 test-elk2.software systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-0.slice has finished shutting down.
Nov 29 10:06:54 test-elk2.software polkitd[1030]: Registered Authentication Agent for unix-process:13216:655441642 (system bus name :1.90614 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKi
Nov 29 10:06:54 test-elk2.software systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: A tool for managing settings stored in the elasticsearch keystore
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: Non-option arguments:
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: command
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: Option             Description
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: ------             -----------
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -E <KeyValuePair>  Configure a setting
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -h, --help         Show help
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -s, --silent       Show minimal output
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -v, --verbose      Show verbose output
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: ERROR: Unknown command [has-passwd]
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: Exception in thread "main" java.lang.IllegalArgumentException: expected one argument specifying path to jvm.options but was [/etc/elasticsearch, /usr/share/elasticsearch
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:59)
Nov 29 10:07:01 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 10:07:01 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 10:07:01 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 10:07:01 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 10:07:01 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:13216:655441642 (system bus name :1.90614, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

Where is the problem? What should I do?

Thanks

2

Hi Farid,

This logs doesnt give much information, can you enable debug logs

Only thing i can see there is jvm.options file missing, might be related to read/write rights of file

3

Thank you FALEN for your response

This is my jvm.options

[root@test-elk2 conf]# cat /etc/elasticsearch/jvm.options
################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/jvm-options.html
## for more information.
##
################################################################



################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
-Xms4g
-Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/heap-size.html
## for more information
##
################################################################


################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

how can I enable debug logs?

4

This is my java -version

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
5

This is what journal -xe shows:

-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 13:15:47 test-elk2.software elasticsearch[24774]: could not find java in bundled JDK at /usr/share/elasticsearch/jdk/bin/java
Nov 29 13:15:47 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 13:15:47 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 13:15:47 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 13:15:47 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 13:15:47 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:24768:656574958 (system bus name :1.90834, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (
Nov 29 13:17:25 test-elk2.software CommAmqpListene[24423]: [CCafException] AmqpComm@[24423]: CommAmqpListener: [CCafException] AmqpCommon::validateStatus() amqp_socket_open_noblock: hostname lookup failed
Nov 29 13:17:56 test-elk2.software polkitd[1030]: Registered Authentication Agent for unix-process:25575:656587913 (system bus name :1.90835 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKi
Nov 29 13:17:56 test-elk2.software systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 13:17:56 test-elk2.software elasticsearch[25581]: could not find java in bundled JDK at /usr/share/elasticsearch/jdk/bin/java
Nov 29 13:17:56 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 13:17:56 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 13:17:56 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 13:17:56 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 13:17:56 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:25575:656587913 (system bus name :1.90835, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (
Nov 29 13:18:29 test-elk2.software polkitd[1030]: Registered Authentication Agent for unix-process:25949:656591139 (system bus name :1.90836 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKi
Nov 29 13:18:29 test-elk2.software systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 13:18:29 test-elk2.software elasticsearch[25955]: could not find java in bundled JDK at /usr/share/elasticsearch/jdk/bin/java
Nov 29 13:18:29 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 13:18:29 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 13:18:29 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 13:18:29 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 13:18:29 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:25949:656591139 (system bus name :1.90836, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
6

Please check your /var/log/elasticsearch directory.

7

Hi @Farid_N

This concerns me greatly

How did you install elasticsearch initially? tar.gz, deb, rpm?

Exactly how did you upgrade?... you can not just piecemeal copy binaries etc.

8

Hi @stephenb
I downloaded the latest elasticsearch.tar.gz
Then extracted it and replace the conf directory in /etc/elasticsearch and copy other content in /usr/share/elasticsearch

It seems could not run with the current JDK installed on my OS

When I rolled back everything and copy the previous version that run before, it works correctly again...