# Cluster goes yellow a few times during the night
**URL:** https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972
**Category:** Elasticsearch
**Created:** [November 24, 2017, 6:24am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972 "2017-11-24T06:24:59Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@McElroy](https://discuss.elastic.co/u/McElroy)
#### Post date: [November 24, 2017, 6:24am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/1 "2017-11-24T06:24:59Z")
Hi,
my elasticsearch cluster goes yellow every once in awhile during the night. It is because it starts to put the shards on the other nodes. Can I do anything to prevent my cluster from going yellow?
```
[2017-11-24T03:08:37,386][INFO][o.e.c.m.MetaDataCreateIndexService] [server1] [admpwd-2017.11.24] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2017-11-24T03:08:43,072][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[admpwd-2017.11.24][2], [admpwd-2017.11.24][1], [admpwd-2017.11.24][0], [admpwd-2017.11.24][4]] ...]).
```
My elasticversion:
```
{
"name" : "server1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "7MDCcLhDQkCoJYkGGwxVOg",
"version" : {
"number" : "5.6.1",
"build_hash" : "667b497",
"build_date" : "2017-09-14T19:22:05.189Z",
"build_snapshot" : false,
"lucene_version" : "6.6.1"
},
"tagline" : "You Know, for Search"
}
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 24, 2017, 7:19am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/2 "2017-11-24T07:19:49Z")
What is wrong with that? Yellow is not critical I mean.
---
### Author:  [@McElroy](https://discuss.elastic.co/u/McElroy)
#### Post date: [November 24, 2017, 7:39am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/3 "2017-11-24T07:39:40Z")
I know that it isn't critical, but I have no idea why this is happening and why it is only happening at night?
The cluster has way more work to do during the day, so why is this only happening at night?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 24, 2017, 7:47am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/4 "2017-11-24T07:47:49Z")
Are you using time-based indices? Do the status change coincide with the creation of new indices?
---
### Author:  [@McElroy](https://discuss.elastic.co/u/McElroy)
#### Post date: [November 24, 2017, 7:53am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/5 "2017-11-24T07:53:06Z")
The indices are created per day. Not really, I do have some status change a little bit after 12pm.
I attached the logs so you see what I mean.
```
[2017-11-24T00:00:07,090][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rsyslog-hob-vmware-2017.11.24][0], [rsyslog-hob-vmware-2017.11.24][4], [rsyslog-hob-vmware-2017.11.24][2], [rsyslog-hob-vmware-2017.11.24][1]] ...]).
[2017-11-24T00:02:30,953][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rsyslog-as4u-sec-2017.11.24][2], [rsyslog-as4u-sec-2017.11.24][4], [rsyslog-as4u-sec-2017.11.24][0], [rsyslog-as4u-sec-2017.11.24][3]] ...]).
[2017-11-24T01:00:11,369][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[dc-2017.11.24][2], [dc-2017.11.24][4], [dc-2017.11.24][1], [dc-2017.11.24][0]] ...]).
[2017-11-24T01:00:22,835][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[ftp-2017.11.24][4], [ftp-2017.11.24][3], [ftp-2017.11.24][2], [ftp-2017.11.24][0]] ...]).
[2017-11-24T01:00:29,230][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rsyslog-lb-2017.11.24][2], [rsyslog-lb-2017.11.24][1], [rsyslog-lb-2017.11.24][4], [rsyslog-lb-2017.11.24][0]] ...]).
[2017-11-24T01:04:23,009][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[history-2017.11.24][0], [history-2017.11.24][2], [history-2017.11.24][4], [history-2017.11.24][3]] ...]).
[2017-11-24T03:08:43,072][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[admpwd-2017.11.24][2], [admpwd-2017.11.24][1], [admpwd-2017.11.24][0], [admpwd-2017.11.24][4]] ...]).
[2017-11-24T07:00:06,229][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[se-tcj-wl-2017-11-24][4], [se-tcj-wl-2017-11-24][2], [se-tcj-wl-2017-11-24][0]] ...]).
[2017-11-24T07:16:01,040][INFO][o.e.c.r.a.AllocationService] [server1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[se-tcj-al-2017-11-24][4], [se-tcj-al-2017-11-24][0], [se-tcj-al-2017-11-24][2], [se-tcj-al-2017-11-24][1]] ...]).
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 24, 2017, 7:57am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/6 "2017-11-24T07:57:42Z")
This is because you create a new index every night.
It allocates first the primaries (yellow status) then it starts to allocate and recover from the primaries the replicas. When done, index becomes green.
If you want to avoid that, you can create in advance the indices (empty ones) so allocation will happen faster IMO.
BTW: don't keep default values (5 shards). That's probably too much. (unsure though depending on your dataset)
---
### Author:  [@McElroy](https://discuss.elastic.co/u/McElroy)
#### Post date: [November 24, 2017, 8:01am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/7 "2017-11-24T08:01:59Z")
I have 4 data nodes, if that's what you mean.
How can I change the default shard behavior?
EDIT:
Is it with that command?
If I want to set it globally, I just ditch the index in that example?
```
curl -XPUT 'localhost:9200/twitter/_settings?pretty' -H 'Content-Type: application/json' -d'
{
"index" : {
"number_of_replicas" : 2
}
}
'
```
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 24, 2017, 8:27am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/8 "2017-11-24T08:27:37Z")
No. I meant that 5 primary shards is probably too much.
Watch this video:
---
### Author:  [@McElroy](https://discuss.elastic.co/u/McElroy)
#### Post date: [November 24, 2017, 8:39am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/9 "2017-11-24T08:39:02Z")
I have to watch that later at the university. Thank you for your help so far.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 24, 2017, 8:47am UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/10 "2017-11-24T08:47:05Z")
Use an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/indices-templates.html) to control the number of replicas when indices are created.
---
### Author:  [@McElroy](https://discuss.elastic.co/u/McElroy)
#### Post date: [November 27, 2017, 12:02pm UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/11 "2017-11-27T12:02:30Z")
Hi,
i wrote a go program which gets the current indices and creates indices for the next day.
If anyone has the same problem as me, feel free to use it, fork it, burn it, mix it or whatever pleases you.
> **[n0r1sk / acei](https://gitlab.com/n0r1sk/acei)**
>
> Automatic Creation of ElasticSearch Indices
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 25, 2017, 12:02pm UTC](https://discuss.elastic.co/t/cluster-goes-yellow-a-few-times-during-the-night/108972/12 "2017-12-25T12:02:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.