Unable to start Elasticsearch 7.6.2 as a service [RHEL 7.5]

Elastic Stack Elasticsearch
1

Hello fellas

I dowloaded and installed the RPM version - and GPG-Key - manually, following the steps given by elastic.co...
installation file: "elasticsearch-7.6.2-x86_64.rpm"

The server that I'm working on have strict security and is behind a proxy that I'm not able to pass to do the yum repo installation version

The installation went fine.

And since it was installed I'm unable to start the elasticsearch.service using

# systemctl start elasticsearch.service

and it always return "Failled state" and shuts down.
When I tried to check the reason looking at the application log it was empty.
It only fill the journalctl with informations regarding the errors.

However, it always works fine when I run the executable, located at /usr/share/elasticsearch/bin/elasticsearch .
I checked the app log and the info that elasticsearch was working fine was there.

So much so that it was possible to install kibana and configure both elastic and kibana.
However, running the executable, even in background, is not the best solution in my case. I need the service running.

Below are the configuration files and logs and, if you need any more, I'll post the ones that you request:

elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: myserver
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: []
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

both (/var/lib/elasticsearch) and (/var/log/elasticsearch) have the permissions 644

2

journalctl --unit elasticsearch :

Apr 30 15:30:17 myserver systemd[1]: Starting Elasticsearch...
Apr 30 15:30:17 myserver elasticsearch[7313]: future versions of Elasticsearch will require Java 11; your Java version from [/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre] does not meet this requirement
Apr 30 15:30:18 myserver elasticsearch[7313]: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Apr 30 15:30:36 myserver elasticsearch[7313]: 2020-04-30 15:30:36,865 main ERROR RollingFileManager (/var/log/elasticsearch/elasticsearch_server.json) java.io.FileNotFoundException: /var/log/elasticsearch/elasticsearch_server.json (Permission denied) java.io.FileNotFoundException: /var/log/elasticsearch/elasticsearch_server.json (Permission denied)
Apr 30 15:30:36 myserver elasticsearch[7313]: at java.io.FileOutputStream.open0(Native Method)
Apr 30 15:30:36 myserver elasticsearch[7313]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Apr 30 15:30:36 myserver elasticsearch[7313]: at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
Apr 30 15:30:36 myserver elasticsearch[7313]: at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Apr 30 15:30:36 myserver elasticsearch[7313]: 2020-04-30 15:30:36,889 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@76f2bbc1] unable to create manager for [/var/log/elasticsearch/elasticsearch_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@68e965f5[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
Apr 30 15:30:36 myserver elasticsearch[7313]: }, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@76f2bbc1] unable to create manager for [/var/log/elasticsearch/elasticsearch_server.json] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@68e965f5[pattern=/var/log/elasticsearch/elasticsearch-%d{yyyy-MM-dd}-%i.json.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
Apr 30 15:30:36 myserver elasticsearch[7313]: }, filePermissions=null, fileOwner=null]]
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Apr 30 15:30:36 myserver elasticsearch[7313]: 2020-04-30 15:30:36,890 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
Apr 30 15:30:36 myserver elasticsearch[7313]: at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
.
.
.
Apr 30 15:30:37 myserver elasticsearch[7313]: 2020-04-30 15:30:37,129 main ERROR Null object returned for RollingFile in Appenders.
Apr 30 15:30:37 myserver elasticsearch[7313]: 2020-04-30 15:30:37,129 main ERROR Unable to locate appender "rolling" for logger config "root"
Apr 30 15:30:38 myserver elasticsearch[7313]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /var/run/elasticsearch
Apr 30 15:30:38 myserver elasticsearch[7313]: Likely root cause: java.nio.file.AccessDeniedException: /var/run/elasticsearch
Apr 30 15:30:38 myserver elasticsearch[7313]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Apr 30 15:30:38 myserver elasticsearch[7313]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Apr 30 15:30:38 myserver elasticsearch[7313]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Apr 30 15:30:38 myserver elasticsearch[7313]: at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
Apr 30 15:30:38 myserver elasticsearch[7313]: at java.nio.file.Files.createDirectory(Files.java:674)
Apr 30 15:30:38 myserver elasticsearch[7313]: at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
Apr 30 15:30:38 myserver elasticsearch[7313]: at java.nio.file.Files.createDirectories(Files.java:767)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.common.PidFile.create(PidFile.java:69)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.common.PidFile.create(PidFile.java:55)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:324)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.cli.Command.main(Command.java:90)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Apr 30 15:30:38 myserver elasticsearch[7313]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Apr 30 15:30:38 myserver elasticsearch[7313]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
Apr 30 15:30:38 myserver systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 30 15:30:38 myserver systemd[1]: Failed to start Elasticsearch.
Apr 30 15:30:38 myserver systemd[1]: Unit elasticsearch.service entered failed state.

[admin@myserver ~]$ java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 8.0.5.35 - pxa6480sr5fp35ifix-20190524_01(SR5 FP35+IJ15979))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190524_417294 (JIT enabled, AOT enabled)
OpenJ9 - 321e430
OMR - 16b77d7
IBM - 72459d3)
JCL - 20190409_01 based on Oracle jdk8u211-b25

$JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre

Note that "/var/log/elasticsearch/elasticsearch.log" only contains the log from the time I started via executable (/usr/share/elasticsearch/bin/elasticsearch) until I ended manually, so I'm not pasting it here...

I already checked the permissions and grant 644 for some of the files which had 700
I didn't change the owner of any of the files.

What do I need to do to make it work?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.