Unable to start Elasticsearch after clean installation (CentOS or Ubuntu)

Check the full logs with journalctl -u elasticsearch.service, the reason why it failed to start would be there

Thanks ikakavas,

That command doesn't seem to show updated entries but I ran sudo systemctl status elasticsearch.service -l and got the following output:

[tedl@localhost ~]$ sudo systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2019-11-18 05:54:14 EST; 3min 54s ago
     Docs: http://www.elastic.co
  Process: 6058 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid (code=exited, status=1/FAILURE)
 Main PID: 6058 (code=exited, status=1/FAILURE)

Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.2.jar:7.4.2]
Nov 18 05:54:14 localhost.localdomain elasticsearch[6058]: ... 6 more
Nov 18 05:54:14 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 18 05:54:14 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 18 05:54:14 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 18 05:54:14 localhost.localdomain systemd[1]: elasticsearch.service failed.

Here's what I get when I run the command you requested ikakakvas:

Nov 17 17:33:39 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: encountered [1] error parsing [/etc/elasticsearch/jvm.options]
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: [1]: encountered improperly formatted JVM option line [es.enforce.bootstrap.checks=true] on line nu
Nov 17 17:33:40 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 17 17:33:40 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 17 17:33:40 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 17 17:33:40 localhost.localdomain systemd[1]: elasticsearch.service failed.
Nov 17 17:34:24 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 17 17:34:25 localhost.localdomain elasticsearch[4791]: encountered [1] error parsing [/etc/elasticsearch/jvm.options]
Nov 17 17:34:25 localhost.localdomain elasticsearch[4791]: [1]: encountered improperly formatted JVM option line [es.enforce.bootstrap.checks=true] on line nu
Nov 17 17:34:25 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 17 17:34:25 localhost.localdomain systemd[1]: Failed to start Elasticsearch.
Nov 17 17:34:25 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 17 17:34:25 localhost.localdomain systemd[1]: elasticsearch.service failed.
Nov 17 17:36:19 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 17 17:36:20 localhost.localdomain elasticsearch[4878]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will 
Nov 17 17:36:26 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:26,978][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [
Nov 17 17:36:27 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:27,026][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [1015.6mb], compress
Nov 17 17:36:27 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:27,030][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [t
Nov 17 17:36:27 localhost.localdomain elasticsearch[4878]: [2019-11-17T17:36:27,031][INFO ][o.e.n.Node               ] [node-1] version[7.4.2], pid[4878], bui
lines 1-39

No entries from today for some reason.

/etc/elasticsearch/elasticsearch.yml content:

# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.200.49
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

/etc/elasticsearch/jvm.options content:

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
# 10-:-XX:+UseG1GC
# 10-:-XX:G1ReservePercent=25
# 10-:-XX:InitiatingHeapOccupancyPercent=30

## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10

## optimizations

# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# explicitly set the stack size
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow

# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dio.netty.allocator.numDirectArenas=0

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT

There's your error right there:

Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: encountered [1] error parsing [/etc/elasticsearch/jvm.options]
Nov 17 17:33:40 localhost.localdomain elasticsearch[4713]: [1]: encountered improperly formatted JVM option line [es.enforce.bootstrap.checks=true] on line nu
Nov 17 17:33:40 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE

Remove the linees.enforce.bootstrap.checks=true that you have added in /etc/elasticsearch/jvm.options

Thanks ikakavas,
That error has been fixed.
Sorry for the unformatted text. I hope it looks better now between Quote tags.
I had kept a copy of the original /etc/elasticsearch/jvm.options file and restored that when I saw the error.
I was trying to enforce bootstrap to see if I could start the service though. Any thoughts on whether that may help or how to do it?
Kind regards,
Ted

Just run the service now and show us the output from the same command I asked before so that we can see what the underlying error is

Hi ikakakvas,
I did just run the command to start the service and the log entries you asked for did not update.
Would recent entries in /var/log/elasticsearch/elasticsearch.log help?

Last entry from /var/log/elasticsearch/elasticsearch.log:

[2019-11-18T05:54:14,885][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.4.2.jar:7.4.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.4.2.jar:7.4.2]
Caused by: org.elasticsearch.ElasticsearchException: Failure running machine learning native code. This could be due to running on an unsupported OS or distribution, missing OS libraries, or a problem with the temp directory. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].
        at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:524) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$9(Node.java:450) ~[elasticsearch-7.4.2.jar:7.4.2]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1621) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:453) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.node.Node.<init>(Node.java:255) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.4.2.jar:7.4.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.4.2.jar:7.4.2]

see Elasticsearch 7.1.1 Failed to create native process factories for Machine Learning

I did find the entries you're looking for ikakakvas:

The log entry was too long so here's a gist link.

I apologize for the frustration in getting you what you need and I greatly appreciate the help!

Thanks ikakavas!

Checking that thread now.

I disabled machine learning and now get an error related to bootstrap checks.

Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: [2019-11-18T06:53:26,116][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-l
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: ERROR: [1] bootstrap checks failed
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.s
Nov 18 06:53:26 localhost.localdomain elasticsearch[6266]: [2019-11-18T06:53:26,167][INFO ][o.e.n.Node               ] [node-1] stopping ...

Sorry the output is truncated. Looking for how to get the text not to truncate.

I am running on very old hardware.

Please go through our documentation , I'm sure you'll find it immensely useful https://www.elastic.co/guide/en/elasticsearch/reference/current/_discovery_configuration_check.html

Hi @theo1

It looks like your cluster is configured with only one node. Because you specified network.host your cluster is going through the bootstrap checks, take a look here.

On the same page you also find an option to set for single-discovery. Please try it with this setting.

1 Like

Thanks ikakavas and MiTschMR.

It's working now. Here's a diff between the original file and the current file so you can see what's changed:

[tedl@localhost ~]$ sudo diff /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.0
17c17
< cluster.name: wazuh-el-log
---
> #cluster.name: my-application
23c23
< node.name: wazuh-el-node1
---
> node.name: node-1
59c59
< http.port: 9200
---
> #http.port: 9200
72c72
< cluster.initial_master_nodes: ["wazuh-el-node1"]
---
> cluster.initial_master_nodes: ["node-1"]
89,92d88
< #
< # Adding entry to get past error
< #
< xpack.ml.enabled: false

The xpack.ml.enabled:false disables Machine Learning. I'm using a very old processor and it seemed to be causing issues with Machine Learning started.

I'm going to try the installation process on a newer system to see if I can have Machine Learning enabled.

Many thanks for all the help!

Ted

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.