ES Cluster goes red at same time every day

Elastic Stack Elasticsearch
1

As title suggests, at 4PM PST (which is 12AM on the boxes that host the cluster) my cluster goes into red state for a brief second. This happens consistently every day. I observed it last week in marvel and it seemed to be red for less than the duration of a 5 second minimum refresh. Our ES clusters in pre production environments exhibit the same behavior.

The cluster does not appear to be losing any requests during this time, but it's still of concern given that 4PM PST is a peek hour for our service. My questions are as follows:

A. is this normal behavior for an ES cluster?
B. if so, is it possible to move this reset to a different time when we know there will be less requests?

2

I imagine this is caused by you creating a new daily index and the cluster blips red while the index allocates. Lots of things can effect the length of this time like your version and the number of indexes.

You can likely move this time but I wouldn't worry about it.