Elasticsearch won't index logs while cluster is red

[hunsw (Attila)]

Elasticsearch version: 6.6
Logstash version: 6.6
SSL is on

After restarting the cluster or even a single node that has some indices with primary 1 replica 0 shard setting, Logstash starts to accumulate events and it looks like they won't arrive into Elasticsearch until the cluster turns at least yellow. (This means at least 20-30 minutes. :/)

Even in a yellow state, the Logstash persistent queue is very slow to empty. (Cc. 1-5 MByte/sec.)

This is happening in a 15 node cluster with 3 Logstashes sending events to 3 Elasticsearch data nodes. I couldn't really find documentation if setting up Logstash to forward events to say 5 or 6 nodes can help or not...

So... is it normal that the ES cluster cannot index logs for a good while after a single node or a cluster restart? How can we help this?

[Preakness]

Hello hunsw,
It's hard to answer your question without knowing the reason the cluster is red or yellow. If primary shards are offline and you have no replicas then I'd say yes what you are seeing is normal.
When your cluster is red or yellow use the explain api to get more information about shards.
For example

curl -u:user:password -XGET  https://elasticsearch_server:9200/_cluster/allocation/explain?pretty

[hunsw (Attila)]

The exact scenario is this: I have 3 Logstashes, sending logs to Elasticsearch data nodes (let's say e1, e2, e3). I have an index (i1) with 3 primary shards (s0-s2) and 0 replica shards.

My question is if either of these cases are normal:

a) e1 is shut down (cluster goes red, because s0 of i1 was sitting there) -> logs start accumulating in the Elasticsearch output queues

b) e4 is shut down and the same happens as in a)

Maybe it's the ES cluster getting a bit overloaded because of the recoveries starting due the node shutdown?
I think this is happening even if I disable shard allocation. (Cluster routing set to none.)

[Christian_Dahlqvist]

If all primary shards of an index are not available you can not index into it. That is expected. If you have no replicas there is no copy Elasticsearch can promote, which means the cluster can not automatically get back to green/yellow state.

[hunsw (Attila)]

I was not clear enough... the i1 index I mentioned above is an old index, the ones I'm indexing into (even while the cluster is red) are 'green', they have all shards online.

[Christian_Dahlqvist]

I do believe indexing is blocked while in red state, so that would be expected.

[DavidTurner]

Elasticsearch doesn't block indexing while the cluster health is red. The cluster-wide health is the health of the least-healthy shard, so it can be red while other indices are yellow or green and are therefore available for writes. @Christian_Dahlqvist do you mean that Logstash stops trying to index into Elasticsearch while the cluster health is red?

[Christian_Dahlqvist]

I believe it just sends bulk requests so as long as no red indices are indexed into it should be fine. If any red index is indexed into anywhere in the pipeline it could however cause back pressure to build up.