Elasticsearch wont run

Hi dears
I have upgraded the Elasticsearch and removed old version and copy all the new Elasticsearch parts in the same directory
(upgrading ES 7.6.2 to 7.15.2)
Unfortunately when I want to start the service with sysV this exception occurs:

[root@test-elk2 tools]# service elasticsearch start
Starting elasticsearch (via systemctl):  Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
                                                           [FAILED]
[root@test-elk2 tools]# journalctl -xe
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: Non-option arguments:
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: command
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: Option             Description
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: ------             -----------
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -E <KeyValuePair>  Configure a setting
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -h, --help         Show help
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -s, --silent       Show minimal output
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: -v, --verbose      Show verbose output
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: ERROR: Unknown command [has-passwd]
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: Exception in thread "main" java.lang.IllegalArgumentException: expected one argument specifying path to jvm.options but was [/etc/elasticsearch, /usr/share/elasticsearch
Nov 29 10:05:09 test-elk2.software elasticsearch[12284]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:59)
Nov 29 10:05:09 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 10:05:09 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 10:05:09 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 10:05:09 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 10:05:09 test-elk2.software systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-0.slice has finished shutting down.
Nov 29 10:06:54 test-elk2.software polkitd[1030]: Registered Authentication Agent for unix-process:13216:655441642 (system bus name :1.90614 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKi
Nov 29 10:06:54 test-elk2.software systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: A tool for managing settings stored in the elasticsearch keystore
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: Non-option arguments:
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: command
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: Option             Description
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: ------             -----------
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -E <KeyValuePair>  Configure a setting
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -h, --help         Show help
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -s, --silent       Show minimal output
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: -v, --verbose      Show verbose output
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: ERROR: Unknown command [has-passwd]
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: Exception in thread "main" java.lang.IllegalArgumentException: expected one argument specifying path to jvm.options but was [/etc/elasticsearch, /usr/share/elasticsearch
Nov 29 10:07:01 test-elk2.software elasticsearch[13222]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:59)
Nov 29 10:07:01 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 10:07:01 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 10:07:01 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 10:07:01 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 10:07:01 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:13216:655441642 (system bus name :1.90614, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) 

Where is the problem? What should I do?

Thanks

Hi Farid,

This logs doesnt give much information, can you enable debug logs

Only thing i can see there is jvm.options file missing, might be related to read/write rights of file

Thank you FALEN for your response

This is my jvm.options

[root@test-elk2 conf]# cat /etc/elasticsearch/jvm.options
################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/jvm-options.html
## for more information.
##
################################################################



################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
-Xms4g
-Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/heap-size.html
## for more information
##
################################################################


################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

how can I enable debug logs?

This is my java -version

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

This is what journal -xe shows:

-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 13:15:47 test-elk2.software elasticsearch[24774]: could not find java in bundled JDK at /usr/share/elasticsearch/jdk/bin/java
Nov 29 13:15:47 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 13:15:47 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 13:15:47 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 13:15:47 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 13:15:47 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:24768:656574958 (system bus name :1.90834, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (
Nov 29 13:17:25 test-elk2.software CommAmqpListene[24423]: [CCafException] AmqpComm@[24423]: CommAmqpListener: [CCafException] AmqpCommon::validateStatus() amqp_socket_open_noblock: hostname lookup failed
Nov 29 13:17:56 test-elk2.software polkitd[1030]: Registered Authentication Agent for unix-process:25575:656587913 (system bus name :1.90835 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKi
Nov 29 13:17:56 test-elk2.software systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 13:17:56 test-elk2.software elasticsearch[25581]: could not find java in bundled JDK at /usr/share/elasticsearch/jdk/bin/java
Nov 29 13:17:56 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 13:17:56 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 13:17:56 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 13:17:56 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 13:17:56 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:25575:656587913 (system bus name :1.90835, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (
Nov 29 13:18:29 test-elk2.software polkitd[1030]: Registered Authentication Agent for unix-process:25949:656591139 (system bus name :1.90836 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKi
Nov 29 13:18:29 test-elk2.software systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 13:18:29 test-elk2.software elasticsearch[25955]: could not find java in bundled JDK at /usr/share/elasticsearch/jdk/bin/java
Nov 29 13:18:29 test-elk2.software systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 29 13:18:29 test-elk2.software systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 13:18:29 test-elk2.software systemd[1]: Unit elasticsearch.service entered failed state.
Nov 29 13:18:29 test-elk2.software systemd[1]: elasticsearch.service failed.
Nov 29 13:18:29 test-elk2.software polkitd[1030]: Unregistered Authentication Agent for unix-process:25949:656591139 (system bus name :1.90836, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)

Please check your /var/log/elasticsearch directory.

1 Like

Hi @Farid_N

This concerns me greatly

How did you install Elasticsearch initially? tar.gz, deb, rpm?

Exactly how did you upgrade?... you can not just piecemeal copy binaries etc.

1 Like

Hi @stephenb
I downloaded the latest elasticsearch.tar.gz
Then extracted it and replace the conf directory in /etc/elasticsearch and copy other content in /usr/share/elasticsearch

It seems could not run with the current JDK installed on my OS

When I rolled back everything and copy the previous version that run before, it works correctly again...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.