# Elasticsearch won't index logs while cluster is red **URL:** \ **Category:** Elasticsearch\ **Created:** [September 5, 2019, 1:30pm UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265 "2019-09-05T13:30:25Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![hunsw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hunsw/32/93637_2.png) [@hunsw](https://discuss.elastic.co/u/hunsw)\ **Post date:** [September 5, 2019, 1:30pm UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/1 "2019-09-05T13:30:25Z")
Elasticsearch version: 6.6 Logstash version: 6.6 SSL is on After restarting the cluster or even a single node that has some indices with primary 1 replica 0 shard setting, Logstash starts to accumulate events and it looks like they won't arrive into Elasticsearch until the cluster turns at least yellow. (This means at least 20-30 minutes. :/) Even in a yellow state, the Logstash persistent queue is very slow to empty. (Cc. 1-5 MByte/sec.) This is happening in a 15 node cluster with 3 Logstashes sending events to 3 Elasticsearch data nodes. I couldn't really find documentation if setting up Logstash to forward events to say 5 or 6 nodes can help or not... So... is it normal that the ES cluster cannot index logs for a good while after a single node or a cluster restart? How can we help this? ---
**Author:** ![Preakness](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preakness/32/35560_2.png) [@Preakness](https://discuss.elastic.co/u/Preakness)\ **Post date:** [September 5, 2019, 1:53pm UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/2 "2019-09-05T13:53:34Z")
Hello hunsw, It's hard to answer your question without knowing the reason the cluster is red or yellow. If primary shards are offline and you have no replicas then I'd say yes what you are seeing is normal. When your cluster is red or yellow use the explain api to get more information about shards. For example ```auto curl -u:user:password -XGET https://elasticsearch_server:9200/_cluster/allocation/explain?pretty ``` ---
**Author:** ![hunsw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hunsw/32/93637_2.png) [@hunsw](https://discuss.elastic.co/u/hunsw)\ **Post date:** [September 6, 2019, 7:05am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/3 "2019-09-06T07:05:29Z")
The exact scenario is this: I have 3 Logstashes, sending logs to Elasticsearch data nodes (let's say e1, e2, e3). I have an index (i1) with 3 primary shards (s0-s2) and 0 replica shards. My question is if either of these cases are normal: a) e1 is shut down (cluster goes red, because s0 of i1 was sitting there) -\> logs start accumulating in the Elasticsearch output queues b) e4 is shut down and the same happens as in a) Maybe it's the ES cluster getting a bit overloaded because of the recoveries starting due the node shutdown? I think this is happening even if I disable shard allocation. 😕 (Cluster routing set to none.) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2019, 7:21am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/4 "2019-09-06T07:21:42Z")
If all primary shards of an index are not available you can not index into it. That is expected. If you have no replicas there is no copy Elasticsearch can promote, which means the cluster can not automatically get back to green/yellow state. ---
**Author:** ![hunsw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hunsw/32/93637_2.png) [@hunsw](https://discuss.elastic.co/u/hunsw)\ **Post date:** [September 6, 2019, 7:25am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/5 "2019-09-06T07:25:49Z")
I was not clear enough... the i1 index I mentioned above is an old index, the ones I'm indexing into (even while the cluster is red) are 'green', they have all shards online. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2019, 8:00am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/6 "2019-09-06T08:00:16Z")
I do believe indexing is blocked while in red state, so that would be expected. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 6, 2019, 8:18am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/7 "2019-09-06T08:18:50Z")
Elasticsearch doesn't block indexing while the cluster health is `red`. The cluster-wide health is the health of the least-healthy shard, so it can be `red` while other indices are `yellow` or `green` and are therefore available for writes. @Christian_Dahlqvist do you mean that Logstash stops trying to index into Elasticsearch while the cluster health is `red`? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2019, 8:24am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/8 "2019-09-06T08:24:04Z")
I believe it just sends bulk requests so as long as no red indices are indexed into it should be fine. If any red index is indexed into anywhere in the pipeline it could however cause back pressure to build up. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2019, 8:24am UTC](https://discuss.elastic.co/t/elasticsearch-wont-index-logs-while-cluster-is-red/198265/9 "2019-10-04T08:24:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.