ElasticSearch - 6.1 - Windows - Not starting

Hello,

I'm using Elastic 6.1.1 on windows , I installed it via the msi setup, the cluster was running fine, we got a unexpected restart of the cluster and now all node display this.

Here the exception log :

Elastic.ProcessHosts.Process.StartupException: Could not start process within (00:02:00): C:\Program Files\Java\jre1.8.0_152\bin\java.exe -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -Xmx20g -Xms20g -Delasticsearch -Des.path.home="d:\homeware\Elastic\Elasticsearch\6.1.1" -Des.path.conf="d:\homeware\Elastic\Elasticsearch\6.1.1\config" -cp "d:\homeware\Elastic\Elasticsearch\6.1.1\lib\elasticsearch-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\elasticsearch-cli-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\HdrHistogram-2.1.9.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\hppc-0.7.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-core-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-dataformat-cbor-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-dataformat-smile-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-dataformat-yaml-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\java-version-checker-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jna-4.4.0-1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\joda-time-2.9.5.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jopt-simple-5.0.2.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jts-1.13.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\log4j-1.2-api-2.9.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\log4j-api-2.9.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\log4j-core-2.9.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-analyzers-common-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-backward-codecs-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-core-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-grouping-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-highlighter-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-join-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-memory-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-misc-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-queries-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-queryparser-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-sandbox-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-spatial-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-spatial-extras-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-spatial3d-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-suggest-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\plugin-cli-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\securesm-1.2.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\snakeyaml-1.17.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\spatial4j-0.6.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\t-digest-3.0.jar" org.elasticsearch.bootstrap.Elasticsearch
   at Elastic.ProcessHosts.Process.ProcessBase.Start()

Here my conf

# Use a descriptive name for your cluster:
cluster.name: redacted
# ------------------------------------ Node ------------------------------------
# Use a descriptive name for the node:
node.name: redacted
# Add custom attributes to the node:
#node.attr.rack: r1
node.max_local_storage_nodes: 1
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: D:\Homeware\Elastic\Elasticsearch\6.1.1\data
# Path to log files:
path.logs: D:\Homeware\Elastic\Elasticsearch\6.1.1\logs
# ----------------------------------- Memory -----------------------------------
# Lock the memory on startup:
bootstrap.memory_lock: true
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
# Elasticsearch performs poorly when the system is swapping the memory.
# ---------------------------------- Network -----------------------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
# If this value is set to localhost, server will be started in DEVELOPMENT mode
# fix a non-localhost adress in order to start server in PRODUCTION mode
network.host: redacted
# Set a custom port for HTTP:
http.port: 9200
# transport activate non-loopback adress needed for PRODUCTION mode
transport.tcp.port: 9300
# For more information, consult the network module documentation.
# --------------------------------- Discovery ----------------------------------
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
discovery.zen.ping.unicast.hosts: ["redacted:9300", "redacted:9300"]
# Increase discover timeout in order to give a better chance to cluster to be formed, default 5s
discovery.zen.ping.unicast.hosts.resolve_timeout: 60s
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
discovery.zen.minimum_master_nodes: 2
# For more information, consult the zen discovery module documentation.
# ---------------------------------- Gateway -----------------------------------
# Block initial recovery after a full cluster restart until N nodes are started:
gateway.recover_after_nodes: 3
gateway.recover_after_time: 5m
# For more information, consult the gateway module documentation.
# ---------------------------------- Various -----------------------------------
# Require explicit names when deleting indices:
#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "/.*/"
# this is the default value, no need to be specified
node.data: true
node.ingest: true
node.master: true

Here my jvm options

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-XX:+HeapDumpOnOutOfMemoryError
-Xmx20g
-Xms20g

Here the logs that I got from the process

[2018-05-28T14:54:33,010][INFO ][o.e.n.Node               ] [redacted] initializing ...
[2018-05-28T14:54:33,088][INFO ][o.e.e.NodeEnvironment    ] [redacted] using [1] data paths, mounts [[(D:)]], net usable_space [704.1gb], net total_space [749.8gb], types [NTFS]
[2018-05-28T14:54:33,088][INFO ][o.e.e.NodeEnvironment    ] [redacted] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-05-28T14:56:05,439][INFO ][o.e.n.Node               ] [redacted] node name [redacted], node ID [MImRKdWQS5WA47WbFJb21w]
[2018-05-28T14:56:05,439][INFO ][o.e.n.Node               ] [redacted] version[6.1.1], pid[20744], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2018-05-28T14:56:05,439][INFO ][o.e.n.Node               ] [redacted] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xmx2g, -Xms2g, -Delasticsearch, -Des.path.home=d:\homeware\Elastic\Elasticsearch\6.1.1, -Des.path.conf=d:\homeware\Elastic\Elasticsearch\6.1.1\config]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [aggs-matrix-stats]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [analysis-common]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [ingest-common]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [lang-expression]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [lang-mustache]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [lang-painless]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [mapper-extras]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [parent-join]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [percolator]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [reindex]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [repository-url]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [transport-netty4]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [tribe]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded plugin [x-pack]
[2018-05-28T14:56:10,392][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/26052] [Main.cc@128] controller (64 bit): Version 6.1.1 (Build c508cf991ee61c) Copyright (c) 2017 Elasticsearch BV

I'm running java : jre1.8.0_152
I got a cluster of 3 nodes.
I tried to increase the service timeout in windows by modifing a registry key, it didn't change the timeout of 2 minutes.

The service still crash and in Event viewer I can see the stack that I posted first saying that

Elastic.ProcessHosts.Process.StartupException: Could not start process within (00:02:00)

Any idea ? When googling this issue I find only two posts:

and another one without answer Can not start Elasticsearch 5.5 service

Help would be reaaaaaaaaaaaaalllllllllllllly appreciated

The first thing you should do IMO is to use the latest version: 6.2.4 and see if that helps.

I can't really afford to do the migration atm, since there is some breaking change between 6.1 and 6.2, do you got any idea ?

Which ones are you concerned with?

I found the issue, it seem there is a C# wrapper with the limitation inside (I installed the msi), it seem to be fix by this pr https://github.com/elastic/windows-installers/pull/164 , I installed a new ES with the zip package (full java it seem, the one with bat..) and I made it target the data path / and log of the old installation.
So far it seem to be working fine, I should have read that the msi one is an experimental one ...

Thanks guys

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.