One ES node of 6-node cluster reports "yellow" state once per day

Elastic Stack Elasticsearch
1

Hi,
we are using Elasticsearch in combination with Graylog and something strange happens. As you can see in config below, We have 70 indicies which are rotated daily. The cluster consists of 6 ES nodes and 2 Graylog nodes. These nodes are monitored through API.

Every day, exactly at midnight, one and only one node reports "yellow" state just for a few seconds and then it is green again. I thought it was related to the rotation, but Iam not sure. I checked and the node is not "master" of the ES cluster.

Does anyone have any idea what it might be?

P.S. There is nothing in the log on the ES node at that time

Thanks!

ES version: 1.7.1
Graylog version: 1.2.1

ES config:

cluster.name: graylog2

# Enable CORS requests because of ELASTIC_HQ plugin
http.cors.enabled: true

# Delay reallocation of cluster by 2 mins when node goes off
# (so we can upgrade ES nodes without reallocation)
index.unassigned.node_left.delayed_timeout: 2m

Graylog config: (only ES related part)

rotation_strategy = time
elasticsearch_max_time_per_index = 1d
elasticsearch_max_number_of_indices = 70
retention_strategy = delete
elasticsearch_shards = 5
elasticsearch_replicas = 1
elasticsearch_index_prefix = graylog2

allow_leading_wildcard_searches = false

allow_highlighting = false

elasticsearch_analyzer = standard
2

It is related to the rotation as the cluster creates a new index and then assigns those shards. Allocation is fast but not instantaneous, so your cluster will go red>yellow>green as it assigns the shards.

3

Thanks for the answer! I thought so too, but I don't know, why only one of the six nodes does it... I thought that status at http://IP:9200/_cluster/health is for whole cluster and not one node. Iam monitoring this link on all nodes but only one reports as yellow.

4

Am I wrong? Do I understand the health status incorrectly?