Cluster goes yellow a few times during the night

Elastic Stack Elasticsearch
1

Hi,

my elasticsearch cluster goes yellow every once in awhile during the night. It is because it starts to put the shards on the other nodes. Can I do anything to prevent my cluster from going yellow?

[2017-11-24T03:08:37,386][INFO ][o.e.c.m.MetaDataCreateIndexService] [server1] [admpwd-2017.11.24] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2017-11-24T03:08:43,072][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[admpwd-2017.11.24][2], [admpwd-2017.11.24][1], [admpwd-2017.11.24][0], [admpwd-2017.11.24][4]] ...]).

My elasticversion:

{
  "name" : "server1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "7MDCcLhDQkCoJYkGGwxVOg",
  "version" : {
    "number" : "5.6.1",
    "build_hash" : "667b497",
    "build_date" : "2017-09-14T19:22:05.189Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}
2

What is wrong with that? Yellow is not critical I mean.

3

I know that it isn't critical, but I have no idea why this is happening and why it is only happening at night?
The cluster has way more work to do during the day, so why is this only happening at night?

4

Are you using time-based indices? Do the status change coincide with the creation of new indices?

5

The indices are created per day. Not really, I do have some status change a little bit after 12pm.
I attached the logs so you see what I mean.

[2017-11-24T00:00:07,090][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rsyslog-hob-vmware-2017.11.24][0], [rsyslog-hob-vmware-2017.11.24][4], [rsyslog-hob-vmware-2017.11.24][2], [rsyslog-hob-vmware-2017.11.24][1]] ...]).
[2017-11-24T00:02:30,953][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rsyslog-as4u-sec-2017.11.24][2], [rsyslog-as4u-sec-2017.11.24][4], [rsyslog-as4u-sec-2017.11.24][0], [rsyslog-as4u-sec-2017.11.24][3]] ...]).
[2017-11-24T01:00:11,369][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[dc-2017.11.24][2], [dc-2017.11.24][4], [dc-2017.11.24][1], [dc-2017.11.24][0]] ...]).
[2017-11-24T01:00:22,835][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[ftp-2017.11.24][4], [ftp-2017.11.24][3], [ftp-2017.11.24][2], [ftp-2017.11.24][0]] ...]).
[2017-11-24T01:00:29,230][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rsyslog-lb-2017.11.24][2], [rsyslog-lb-2017.11.24][1], [rsyslog-lb-2017.11.24][4], [rsyslog-lb-2017.11.24][0]] ...]).
[2017-11-24T01:04:23,009][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[history-2017.11.24][0], [history-2017.11.24][2], [history-2017.11.24][4], [history-2017.11.24][3]] ...]).
[2017-11-24T03:08:43,072][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[admpwd-2017.11.24][2], [admpwd-2017.11.24][1], [admpwd-2017.11.24][0], [admpwd-2017.11.24][4]] ...]).
[2017-11-24T07:00:06,229][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[se-tcj-wl-2017-11-24][4], [se-tcj-wl-2017-11-24][2], [se-tcj-wl-2017-11-24][0]] ...]).
[2017-11-24T07:16:01,040][INFO ][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[se-tcj-al-2017-11-24][4], [se-tcj-al-2017-11-24][0], [se-tcj-al-2017-11-24][2], [se-tcj-al-2017-11-24][1]] ...]).
6

This is because you create a new index every night.

It allocates first the primaries (yellow status) then it starts to allocate and recover from the primaries the replicas. When done, index becomes green.

If you want to avoid that, you can create in advance the indices (empty ones) so allocation will happen faster IMO.

BTW: don't keep default values (5 shards). That's probably too much. (unsure though depending on your dataset)

7

I have 4 data nodes, if that's what you mean.
How can I change the default shard behavior?

EDIT:
Is it with that command?
If I want to set it globally, I just ditch the index in that example?

curl -XPUT 'localhost:9200/twitter/_settings?pretty' -H 'Content-Type: application/json' -d'
{
     "index" : {
         "number_of_replicas" : 2
     }
}
'
8

No. I meant that 5 primary shards is probably too much.

Watch this video:

9

I have to watch that later at the university. Thank you for your help so far.

10

Use an index template to control the number of replicas when indices are created.

11

Hi,

i wrote a go program which gets the current indices and creates indices for the next day.
If anyone has the same problem as me, feel free to use it, fork it, burn it, mix it or whatever pleases you.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.