ElasticSearch installation stopped at indices into cluster_state

minn · October 22, 2018, 7:20am

I am facing same issue like (https://github.com/elastic/elasticsearch/issues/33075), any suggestion?

$ ./elasticsearch
[2018-10-22T07:14:57,429][INFO ][o.e.n.Node [2018-10-22T07:14:57,491][INFO ][o.e.e.NodeEnvironment [2018-10-22T07:14:57,491][INFO ][o.e.e.NodeEnvironment [2018-10-22T07:14:57,493][INFO ][o.e.n.Node [2018-10-22T07:14:57,493][INFO ][o.e.n.Node [2018-10-22T07:14:57,493][INFO ][o.e.n.Node [2018-10-22T07:14:59,118][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,118][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,119][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,119][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,119][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,119][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,120][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,120][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,120][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,120][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService [2018-10-22T07:14:59,121][INFO ][o.e.p.PluginsService ] [] initializing ...
] [tICd8KV] using [1] data paths, mounts [[/ (/dev/mapper/turnkey-root)]], net usable_space [13gb], net total_space [16.6gb], types [ext4]
] [tICd8KV] heap size [1007.3mb], compressed ordinary object pointers [true]
] [tICd8KV] node name derived from node ID [tICd8KVbR3OD0c7Qe2D6_w]; set [node.name] to override
] [tICd8KV] version[6.4.2], pid[13993], build[default/tar/04711c2/2018-09-26T13:34:09.098244Z], OS[Linux/4.9.0-8-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
] [tICd8KV] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.OfLoRSmo, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/ElasticSearch/elasticsearch-6.4.2, -Des.path.conf=/ElasticSearch/elasticsearch-6.4.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
] [tICd8KV] loaded module [aggs-matrix-stats]
] [tICd8KV] loaded module [analysis-common]
] [tICd8KV] loaded module [ingest-common]
] [tICd8KV] loaded module [lang-expression]
] [tICd8KV] loaded module [lang-mustache]
] [tICd8KV] loaded module [lang-painless]
] [tICd8KV] loaded module [mapper-extras]
] [tICd8KV] loaded module [parent-join]
] [tICd8KV] loaded module [percolator]
] [tICd8KV] loaded module [rank-eval]
] [tICd8KV] loaded module [reindex]
] [tICd8KV] loaded module [repository-url]
] [tICd8KV] loaded module [transport-netty4]
] [tICd8KV] loaded module [tribe]
] [tICd8KV] loaded module [x-pack-core]
] [tICd8KV] loaded module [x-pack-deprecation]
] [tICd8KV] loaded module [x-pack-graph]

...
[2018-10-22T07:14:59,123][INFO ][o.e.p.PluginsService ] [tICd8KV] loaded module [x-pack-sql]
[2018-10-22T07:14:59,123][INFO ][o.e.p.PluginsService ] [tICd8KV] loaded module [x-pack-upgrade]
[2018-10-22T07:14:59,123][INFO ][o.e.p.PluginsService ] [tICd8KV] loaded module [x-pack-watcher]
[2018-10-22T07:14:59,124][INFO ][o.e.p.PluginsService ] [tICd8KV] no plugins loaded
[2018-10-22T07:15:03,293][INFO ][o.e.x.s.a.s.FileRolesStore] [tICd8KV] parsed [0] roles from file [/ElasticSearch/elasticsearch-6.4.2/config/roles.yml]
[2018-10-22T07:15:03,819][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/14049] [Main.cc@109] controller (64 bit): Version 6.4.2 (Build 660eefe6f2ea55) Copyright (c) 2018 Elasticsearch BV
[2018-10-22T07:15:04,165][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-10-22T07:15:04,379][INFO ][o.e.d.DiscoveryModule ] [tICd8KV] using discovery type [zen]
[2018-10-22T07:15:05,202][INFO ][o.e.n.Node ] [tICd8KV] initialized
[2018-10-22T07:15:05,202][INFO ][o.e.n.Node ] [tICd8KV] starting ...
[2018-10-22T07:15:05,349][INFO ][o.e.t.TransportService ] [tICd8KV] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-10-22T07:15:05,408][WARN ][o.e.b.BootstrapChecks ] [tICd8KV] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-10-22T07:15:08,460][INFO ][o.e.c.s.MasterService ] [tICd8KV] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {tICd8KV}{tICd8KVbR3OD0c7Qe2D6_w}{SxJHRZzURNulilvqW7A6KA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4148072448, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-10-22T07:15:08,465][INFO ][o.e.c.s.ClusterApplierService] [tICd8KV] new_master {tICd8KV}{tICd8KVbR3OD0c7Qe2D6_w}{SxJHRZzURNulilvqW7A6KA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4148072448, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {tICd8KV}{tICd8KVbR3OD0c7Qe2D6_w}{SxJHRZzURNulilvqW7A6KA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4148072448, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-10-22T07:15:08,540][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [tICd8KV] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-10-22T07:15:08,540][INFO ][o.e.n.Node ] [tICd8KV] started
[2018-10-22T07:15:08,758][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [tICd8KV] Failed to clear cache for realms [[]]
[2018-10-22T07:15:08,804][INFO ][o.e.l.LicenseService ] [tICd8KV] license [8fcf289d-07d7-4b46-b60c-b4d007dba19e] mode [basic] - valid
[2018-10-22T07:15:08,821][INFO ][o.e.g.GatewayService ] [tICd8KV] recovered [0] indices into cluster_state

Christian_Dahlqvist · October 22, 2018, 7:56am

There is no indication of any error in the logs you provided, so I am not sure what issue you are having. Please put some effort into describing what exactly the problem is, because it is not clear from your post.

minn · October 22, 2018, 8:03am

My question is ./elasticsearch is stopping at indices into cluster_state , is it something wrong or normal? (I am just start ES today as beginner)

Christian_Dahlqvist · October 22, 2018, 8:08am

I do not see anything wrong. Can you access the HTTP API after it has started?

minn · October 22, 2018, 8:13am

I am trying to install APM server as well. Logs as below

2018-10-22T08:10:20.030Z	INFO	[http_client]	beater/client.go:49	HTTP Server ready
2018-10-22T08:10:20.031Z	INFO	[onboarding]	beater/onboarding.go:36	Publishing onboarding document
2018-10-22T08:10:21.031Z	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://localhost:9200))
2018-10-22T08:10:22.033Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2018-10-22T08:10:22.033Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s)
2018-10-22T08:10:24.034Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2018-10-22T08:10:24.034Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 2 reconnect attempt(s)
2018-10-22T08:10:28.034Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2018-10-22T08:10:28.034Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 3 reconnect attempt(s)
2018-10-22T08:10:36.035Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2018-10-22T08:10:36.035Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 4 reconnect attempt(s)
2018-10-22T08:10:52.035Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2018-10-22T08:10:52.035Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 5 reconnect attempt(s)
2018-10-22T08:11:24.036Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2018-10-22T08:11:24.036Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 6 reconnect attempt(s)

Firewall allowed 9200 9300 80 443
Cannot access https://APMserverIP:9200/

Christian_Dahlqvist · October 22, 2018, 8:15am

Elasticsearch by default binds to localhost, so will not be accessible from another host. You will need to update the Elasticsearch configuration to change that or locate the APM server on the same host.

minn · October 22, 2018, 8:18am

my config/elasticsearch.yml is

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: elk-app
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: my-apm-hostname
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /esdata/data
#
# Path to log files:
#
#path.logs: /esdata/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#bootstrap.system_call_filter: false
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.100.100
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

minn · October 22, 2018, 8:19am

ElasticSearch and APM server are in the same host

Christian_Dahlqvist · October 22, 2018, 8:28am

I am confused. Your Elasticsearch config seems to bind to a specific interface while your log file indicates this is not the case.

I would recommend you start Elasticsearch and then make sure you can connect to it from the host through the HTTP interface, e.g. using curl. Then configure the APM server to use this interface.

If there are problems, look at the Elasticsearch log files for clues.

system · November 19, 2018, 8:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fresh install stuck Elasticsearch	6	2984	May 10, 2018
Elastic cluster keeps crashing/hanging Elasticsearch	4	4087	July 28, 2017
Set cluster Elasticsearch	12	1608	August 27, 2018
Elasticsearch does not start Elasticsearch	18	2610	December 8, 2017
Elasticsearch Cluster Status is RED Elasticsearch elastic-stack-monitoring	12	708	June 29, 2021

ElasticSearch installation stopped at indices into cluster_state

Related topics