we are using Elasticsearch in combination with Graylog and something strange happens. As you can see in config below, We have 70 indicies which are rotated daily. The cluster consists of 6 ES nodes and 2 Graylog nodes. These nodes are monitored through API.
Every day, exactly at midnight, one and only one node reports "yellow" state just for a few seconds and then it is green again. I thought it was related to the rotation, but Iam not sure. I checked and the node is not "master" of the ES cluster.
Does anyone have any idea what it might be?
P.S. There is nothing in the log on the ES node at that time
ES version: 1.7.1
Graylog version: 1.2.1
cluster.name: graylog2 # Enable CORS requests because of ELASTIC_HQ plugin http.cors.enabled: true # Delay reallocation of cluster by 2 mins when node goes off # (so we can upgrade ES nodes without reallocation) index.unassigned.node_left.delayed_timeout: 2m
Graylog config: (only ES related part)
rotation_strategy = time elasticsearch_max_time_per_index = 1d elasticsearch_max_number_of_indices = 70 retention_strategy = delete elasticsearch_shards = 5 elasticsearch_replicas = 1 elasticsearch_index_prefix = graylog2 allow_leading_wildcard_searches = false allow_highlighting = false elasticsearch_analyzer = standard