ElasticSearch - 6.1 - Windows - Not starting


(Frédéric Ho Can Sung) #1

Hello,

I'm using Elastic 6.1.1 on windows , I installed it via the msi setup, the cluster was running fine, we got a unexpected restart of the cluster and now all node display this.

Here the exception log :

Elastic.ProcessHosts.Process.StartupException: Could not start process within (00:02:00): C:\Program Files\Java\jre1.8.0_152\bin\java.exe -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -Xmx20g -Xms20g -Delasticsearch -Des.path.home="d:\homeware\Elastic\Elasticsearch\6.1.1" -Des.path.conf="d:\homeware\Elastic\Elasticsearch\6.1.1\config" -cp "d:\homeware\Elastic\Elasticsearch\6.1.1\lib\elasticsearch-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\elasticsearch-cli-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\HdrHistogram-2.1.9.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\hppc-0.7.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-core-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-dataformat-cbor-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-dataformat-smile-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jackson-dataformat-yaml-2.8.10.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\java-version-checker-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jna-4.4.0-1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\joda-time-2.9.5.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jopt-simple-5.0.2.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\jts-1.13.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\log4j-1.2-api-2.9.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\log4j-api-2.9.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\log4j-core-2.9.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-analyzers-common-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-backward-codecs-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-core-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-grouping-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-highlighter-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-join-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-memory-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-misc-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-queries-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-queryparser-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-sandbox-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-spatial-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-spatial-extras-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-spatial3d-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\lucene-suggest-7.1.0.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\plugin-cli-6.1.1.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\securesm-1.2.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\snakeyaml-1.17.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\spatial4j-0.6.jar;d:\homeware\Elastic\Elasticsearch\6.1.1\lib\t-digest-3.0.jar" org.elasticsearch.bootstrap.Elasticsearch
   at Elastic.ProcessHosts.Process.ProcessBase.Start()

Here my conf

# Use a descriptive name for your cluster:
cluster.name: redacted
# ------------------------------------ Node ------------------------------------
# Use a descriptive name for the node:
node.name: redacted
# Add custom attributes to the node:
#node.attr.rack: r1
node.max_local_storage_nodes: 1
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
path.data: D:\Homeware\Elastic\Elasticsearch\6.1.1\data
# Path to log files:
path.logs: D:\Homeware\Elastic\Elasticsearch\6.1.1\logs
# ----------------------------------- Memory -----------------------------------
# Lock the memory on startup:
bootstrap.memory_lock: true
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
# Elasticsearch performs poorly when the system is swapping the memory.
# ---------------------------------- Network -----------------------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
# If this value is set to localhost, server will be started in DEVELOPMENT mode
# fix a non-localhost adress in order to start server in PRODUCTION mode
network.host: redacted
# Set a custom port for HTTP:
http.port: 9200
# transport activate non-loopback adress needed for PRODUCTION mode
transport.tcp.port: 9300
# For more information, consult the network module documentation.
# --------------------------------- Discovery ----------------------------------
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
discovery.zen.ping.unicast.hosts: ["redacted:9300", "redacted:9300"]
# Increase discover timeout in order to give a better chance to cluster to be formed, default 5s
discovery.zen.ping.unicast.hosts.resolve_timeout: 60s
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
discovery.zen.minimum_master_nodes: 2
# For more information, consult the zen discovery module documentation.
# ---------------------------------- Gateway -----------------------------------
# Block initial recovery after a full cluster restart until N nodes are started:
gateway.recover_after_nodes: 3
gateway.recover_after_time: 5m
# For more information, consult the gateway module documentation.
# ---------------------------------- Various -----------------------------------
# Require explicit names when deleting indices:
#action.destructive_requires_name: true
http.cors.enabled: true
http.cors.allow-origin: "/.*/"
# this is the default value, no need to be specified
node.data: true
node.ingest: true
node.master: true

Here my jvm options

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-XX:+HeapDumpOnOutOfMemoryError
-Xmx20g
-Xms20g

(Frédéric Ho Can Sung) #2

Here the logs that I got from the process

[2018-05-28T14:54:33,010][INFO ][o.e.n.Node               ] [redacted] initializing ...
[2018-05-28T14:54:33,088][INFO ][o.e.e.NodeEnvironment    ] [redacted] using [1] data paths, mounts [[(D:)]], net usable_space [704.1gb], net total_space [749.8gb], types [NTFS]
[2018-05-28T14:54:33,088][INFO ][o.e.e.NodeEnvironment    ] [redacted] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-05-28T14:56:05,439][INFO ][o.e.n.Node               ] [redacted] node name [redacted], node ID [MImRKdWQS5WA47WbFJb21w]
[2018-05-28T14:56:05,439][INFO ][o.e.n.Node               ] [redacted] version[6.1.1], pid[20744], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_152/25.152-b16]
[2018-05-28T14:56:05,439][INFO ][o.e.n.Node               ] [redacted] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xmx2g, -Xms2g, -Delasticsearch, -Des.path.home=d:\homeware\Elastic\Elasticsearch\6.1.1, -Des.path.conf=d:\homeware\Elastic\Elasticsearch\6.1.1\config]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [aggs-matrix-stats]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [analysis-common]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [ingest-common]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [lang-expression]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [lang-mustache]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [lang-painless]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [mapper-extras]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [parent-join]
[2018-05-28T14:56:07,158][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [percolator]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [reindex]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [repository-url]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [transport-netty4]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded module [tribe]
[2018-05-28T14:56:07,173][INFO ][o.e.p.PluginsService     ] [redacted] loaded plugin [x-pack]
[2018-05-28T14:56:10,392][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/26052] [Main.cc@128] controller (64 bit): Version 6.1.1 (Build c508cf991ee61c) Copyright (c) 2017 Elasticsearch BV

I'm running java : jre1.8.0_152
I got a cluster of 3 nodes.
I tried to increase the service timeout in windows by modifing a registry key, it didn't change the timeout of 2 minutes.

The service still crash and in Event viewer I can see the stack that I posted first saying that

Elastic.ProcessHosts.Process.StartupException: Could not start process within (00:02:00)

Any idea ? When googling this issue I find only two posts:

and another one without answer Can not start Elasticsearch 5.5 service

Help would be reaaaaaaaaaaaaalllllllllllllly appreciated


(David Pilato) #3

The first thing you should do IMO is to use the latest version: 6.2.4 and see if that helps.


(Frédéric Ho Can Sung) #4

I can't really afford to do the migration atm, since there is some breaking change between 6.1 and 6.2, do you got any idea ?


(David Pilato) #5

Which ones are you concerned with?


(Frédéric Ho Can Sung) #6

I found the issue, it seem there is a C# wrapper with the limitation inside (I installed the msi), it seem to be fix by this pr https://github.com/elastic/windows-installers/pull/164 , I installed a new ES with the zip package (full java it seem, the one with bat..) and I made it target the data path / and log of the old installation.
So far it seem to be working fine, I should have read that the msi one is an experimental one ...

Thanks guys


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.