Elastic search getting timeout

Elastic Stack Elasticsearch
1

Here is the error i'm getting, I have no clue about this error. Can anyone help me why this is erroring out.

2017-01-04 17:19:24,207][WARN ][rest.suppressed ] /logs-2017.01.05/logEvent Params: {index=logs-2017.01.05, type=logEvent}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

2

What version?
What does your config look like?

3

@warkolm i'm using 2.3.3

 # ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
 cluster.name: elk
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
 node.name: dt1esp01p03
 node.master: true 
 node.data: true
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
 path.data: e:\data
#
# Path to log files:
#
 path.logs: e:\logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
 network.host: dt1esp01p03
#
# Set a custom port for HTTP:
#
 http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
 discovery.zen.ping.unicast.hosts: ["dt1esp01p02", "dt1esp01p01", "dt1esp01p03"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
 discovery.zen.minimum_master_nodes: 2
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true
4

You need to remove the spaces in front of all your config lines.

5

I dont think its a space issue. It was working fine from 5 months until last week

7

it looks like your master node is out of service. is your cluster is green or at least yellow?

8

The cluster looks in red

9

What is the output from the cat nodes API?

10

@Christian_Dahlqvist @wenpos http://localhost:9200/_cat/nodes is loading, but does not show any stats of
nodes. I have attached the cluster health

11

Here is the _cat/nodes output
10.90.88.16 10.90.88.16 67 63 -1.00 d * dt1esp01p01
10.90.88.11 10.90.88.11 74 68 -1.00 d m dt1esp01p03
10.90.88.17 10.90.88.17 35 60 -1.00 d m dt1esp01p02

12

all the shards recovery ？the search action may based on those shards.

13

That looks like a very large number of shards for a 3 node cluster. What is your average shard size? How much heap does each node have?

14

Shard size is 5 per index with one replica

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.