New indexes takes a long time to allocate its new shards

Elastic Stack Elasticsearch
1

Hey Guys,

Got a question, the last 3 days (Under heavy sales volume) our index's via logstash rotate at GMT midnight.

We are noticing that it is taking +20 minutes for the new indexes to get allocated to a Node which during that time the Cluster is in a RED state

I have already adjusted my "Indexing" queue to
indices.memory.index_buffer_size: 70%

because we are consuming about 200GB a day on this cluster and searches are comparatively light

What should I look for that would influence SHARE slowness in creation besides system load?

BTW, after the new indexes are create I have a total of ~1600 Shards (some indexes are small) maybe I should merge some of them and reduce the shard count instead of increasing # of nodes?

2

That's odd, new indices should be created very quickly as they have no data.

How many shards are you creating for the new indices, how many replicas?
How big is your exisiting cluster state; how many existing indices, shards and replicas; how many nodes?

3

Nuts just noticed some of the servers are GMT and some are EDT ick got to fix that

its 9 nodes - each one on its own system ( 8 cors 16GB ram 150GB for data disk )
( at 8GB Heap, 150GB disk space each)
100 index ( 5 shard with 1 replication) ,

SAR

02:12:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:13:01 AM     all     54.01      0.00      6.13      0.74      0.00     39.12
02:14:01 AM     all     61.54      0.00      5.98      0.17      0.00     32.32
02:15:01 AM     all     55.65      0.00      6.11      0.20      0.00     38.04
02:16:01 AM     all     54.54      0.00      5.59      0.28      0.00     39.58
02:17:01 AM     all     70.41      0.00      5.55      0.17      0.00     23.86
02:18:01 AM     all     56.08      0.00      6.81      1.04      0.00     36.07
02:19:02 AM     all     56.93      0.00      6.13      0.27      0.00     36.67
02:20:01 AM     all     59.85      0.00      6.16      0.25      0.00     33.75
02:21:01 AM     all     58.47      0.00      6.02      0.24      0.00     35.27
Average:        all     43.77      0.00      5.61      0.40      0.00     50.22
4

Oh and the HEAP is only 4GB out of the 8GB Total