I've installed Elasticsearch on a clean, new, updated build of CentOS (7.7.1908). This installation is in support of a Wazuh installation using instructions found here:
https://documentation.wazuh.com/3.10/installation-guide/installing-elastic-stack/elastic_server_rpm.html
When I get through the installation process and I'm on step 4. Enable and Start the Elasticsearch Service, the service errors out with very little additional information. Something like service exited with error - error.
I've created a gist with full output from the elasticsearch.log file:
I've tried on a clean build of Ubuntu 18.04 and ran into the same issue.
This CentOS instance is running on as a VM on Proxmox 5.4-13.
It has 2.0 Gigs of RAM and 50 Gig's of hard drive space total. 47 Gig's are free.
Please let me know what other information I can provide to help you help me!
Kind regards,
Ted
Here are the steps so you don't have to follow the link:
Preparation
- Add the Elastic repository and its GPG key:
rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elastic.repo << EOF [elasticsearch-7.x] name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF
Elasticsearch
Elasticsearch is a highly scalable full-text search and analytics engine. For more information, please see Elasticsearch.
- Install the Elasticsearch package:
yum install elasticsearch-7.4.2
- Elasticsearch will only listen on the loopback interface (localhost) by default. Configure Elasticsearch to listen to a non-loopback address by editing the file
/etc/elasticsearch/elasticsearch.ymland uncommenting the settingnetwork.host. Change the value to the IP you want to bind it to:
-
network.host: <elasticsearch_ip>
-
Further configuration will be necessary after changing the
network.hostoption. Add or edit (if commented) the following lines in the file/etc/elasticsearch/elasticsearch.yml:
-
node.name: <node_name> cluster.initial_master_nodes: ["<node_name>"]
-
Enable and start the Elasticsearch service:
- For Systemd:
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
- For SysV Init:
chkconfig --add elasticsearch
service elasticsearch start
Still searching for info on this. [edit]I'm ruinning this as a single node of Elasticsearch.
Here's the message from when I try to start it:
[tedl@localhost ~]$ sudo systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2019-11-17 17:37:04 EST; 59s ago
Docs: http://www.elastic.co
Process: 4878 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid (code=exited, status=1/FAILURE)
Main PID: 4878 (code=exited, status=1/FAILURE)
Nov 17 17:37:04 localhost.localdomain elasticsearch[4878]: at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 17 17:37:04 localhost.localdomain elasticsearch[4878]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 17 17:37:04 localhost.localdomain elasticsearch[4878]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 17 17:37:04 localhost.localdomain elasticsearch[4878]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 17 17:37:04 localhost.localdomain elasticsearch[4878]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 17 17:37:04 localhost.localdomain elasticsearch[4878]: ... 6 more
Nov 17 17:37:04 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 17 17:37:04 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 17 17:37:04 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 17 17:37:04 localhost.localdomain systemd[1]: elasticsearch.service failed.
I'll reply if I find a solution that works.
Check the full logs with journalctl -u elasticsearch.service, the reason why it failed to start would be there
Thanks ikakavas,
That command doesn't seem to show updated entries but I ran sudo systemctl status elasticsearch.service -l and got the following output:
[tedl@localhost ~]$ sudo systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2019-11-18 05:54:14 EST; 3min 54s ago
Docs: http://www.elastic.co
Process: 6058 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid (code=exited, status=1/FAILURE)
Main PID: 6058 (code=exited, status=1/FAILURE)
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: ... 6 more
Nov 18 05:54:14 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 18 05:54:14 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 18 05:54:14 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 18 05:54:14 localhost.localdomain systemd[1]: elasticsearch.service failed.
Here's what I get when I run the command you requested ikakakvas:
Nov 17 17:33:39 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: encountered [1] error parsing [/etc/elasticsearch/jvm.options]
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: [1]: encountered improperly formatted JVM option line [es.enforce.bootstrap.checks=true] on line nu
Nov 17 17:33:40 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 17 17:33:40 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 17 17:33:40 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 17 17:33:40 localhost.localdomain systemd[1]: elasticsearch.service failed.
Nov 17 17:34:24 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 17 17:34:25 localhost.localdomain elasticsearch[4791]: encountered [1] error parsing [/etc/elasticsearch/jvm.options]
Nov 17 17:34:25 localhost.localdomain elasticsearch[4791]: [1]: encountered improperly formatted JVM option line [es.enforce.bootstrap.checks=true] on line nu
Nov 17 17:34:25 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 17 17:34:25 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 17 17:34:25 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 17 17:34:25 localhost.localdomain systemd[1]: elasticsearch.service failed.
Nov 17 17:36:19 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 17 17:36:20 localhost.localdomain elasticsearch[4878]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will
Nov 17 17:36:26 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:26,978][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [
Nov 17 17:36:27 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:27,026][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1015.6mb], compress
Nov 17 17:36:27 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:27,030][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [t
Nov 17 17:36:27 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:27,031][INFO ][o.e.n.Node ] [node-1] version[7.4.2], pid[4878], bui
lines 1-39
No entries from today for some reason.
/etc/elasticsearch/elasticsearch.yml content:
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.200.49
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: truePlease don't post unformatted code, logs, or configuration as it's very hard to read.
Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.
It would be great if you could update your post to solve this.
/etc/elasticsearch/jvm.options content:
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms1g
-Xmx1g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
# 10-:-XX:+UseG1GC
# 10-:-XX:G1ReservePercent=25
# 10-:-XX:InitiatingHeapOccupancyPercent=30
## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# explicitly set the stack size
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dio.netty.allocator.numDirectArenas=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPATThere's your error right there:
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: encountered [1] error parsing [/etc/elasticsearch/jvm.options]
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: [1]: encountered improperly formatted JVM option line [es.enforce.bootstrap.checks=true] on line nu
Nov 17 17:33:40 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Remove the linees.enforce.bootstrap.checks=true that you have added in /etc/elasticsearch/jvm.options
Thanks ikakavas,
That error has been fixed.
Sorry for the unformatted text. I hope it looks better now between Quote tags.
I had kept a copy of the original /etc/elasticsearch/jvm.options file and restored that when I saw the error.
I was trying to enforce bootstrap to see if I could start the service though. Any thoughts on whether that may help or how to do it?
Kind regards,
Ted
Just run the service now and show us the output from the same command I asked before so that we can see what the underlying error is
Hi ikakakvas,
I did just run the command to start the service and the log entries you asked for did not update.
Would recent entries in /var/log/elasticsearch/elasticsearch.log help?
Last entry from /var/log/elasticsearch/elasticsearch.log:
[2019-11-18T05:54:14,885][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.2.jar:7.4.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.2.jar:7.4.2]
Caused by: org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].
at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:524) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$9(Node.java:450) ~[elasticsearch-7.4.2.jar:7.4.2]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1621) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:453) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.2.jar:7.4.2]I did find the entries you're looking for ikakakvas:
The log entry was too long so here's a gist link.
I apologize for the frustration in getting you what you need and I greatly appreciate the help!
Thanks ikakavas!
Checking that thread now.
I disabled machine learning and now get an error related to bootstrap checks.
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: [2019-11-18T06:53:26,116][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-l
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: ERROR: [1] bootstrap checks failed
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.s
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: [2019-11-18T06:53:26,167][INFO ][o.e.n.Node ] [node-1] stopping ...
Sorry the output is truncated. Looking for how to get the text not to truncate.
I am running on very old hardware.
Please go through our documentation , I'm sure you'll find it immensely useful https://www.elastic.co/guide/en/elasticsearch/reference/current/_discovery_configuration_check.html