Node local stats slow 3 nodes cluster

Elastic Stack Elasticsearch
1

Lately I noticed that /_nodes/_local/stats command taking my cluster nodes more than 4s to retrieve stats and the cluster seems to be ok. cpu \ load 30% and heap 60-65% avg.

Can you think about reason of why this happen?

In addition, there is anyway to get these stats faster?

Thanks
D.

2

What version are you on? What OS, JVM?

3

How many indexes and shards does the cluster contain?

4

ES version 2.1.1
Ubuntu 14.04
java version "1.8.0_72"
3,725 indices
17,525 shards

5

Woah. That's a lot. How many nodes?

6

3 nodes - 8GB

7

You're excessively over sharded, you need to fix that.

8

Okay. 6000 shards for an 8 GB node is way way way too much. This is something you need to address. Given the fixed per-shard memory overhead I'm surprised your cluster hasn't collapsed.

To answer your original question , yes, node stats will take a very long time with so much cluster metadata. There's not much to do about that except reduce the amount of metadata.

What kind of indexes do you have? Time-series indexes? Multiple such series? Five shards per index?

9

I have 3 shards + 2 replicas per index
and currently the indices are daily with retention of 90d after that we close them.

The indices are only for logs, from many services \ applications

10

Why?

How much data does all this cover?

11

How many index series? One per application? How many applications? How big is each such (daily) index?

12

Thanks guys for the help and for the ~ numbers I should expect in my cluster.

I almost finish to merge the daily indices to weekly (I hope to see something more build in to reindex weekly to monthly for X period of time)

Now my cluster has 350-400 active indices + 2500~ active shards and it behaves as expected.

May you have any other good tips?

Thanks
D.