Performance degradation

[Burga]

Hi Guys ,

I'm a new in EK , need your help

I need to do performance review due to the slowness of the system

How would you suggest to do this ? and what would you say regarding the amount of servers , do you suggest to add more hosts , and if yes how many ?

I noticed that one of the shard nodes has a half number of shards (2000) of the other nodes (4020) , maybe this is the issue ?

no of shards nodes: 9

logstash nodes : 2

no shards :20,620

documents: 8,977,282,120

Disk Usage: 6.5 T

[Christian_Dahlqvist]

It looks like your average shard size is just over 300MB in size, which is very, very small. Having and querying large number of small shards cause bad performance. I would recommend that you read this blog post and look to reduce the number of shards in the cluster by at least a factor of 10.

It would also help if you could let us know which version you are using and what the specification of your nodes are.

[Burga]

HI , thanks ,

"version" : {
"number" : "6.3.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "eb782d0",
"build_date" : "2018-06-29T21:59:26.107521Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},

hosts specifications:

CPU : 4 (2.20GHz)
Memory : 60G
Storage : 1T

Best Regards.

[Christian_Dahlqvist]

What is flow? Indexing? Querying?

What type of storage are you using? Local SSDs?

[Burga]

Sorry ,

The storage is NFS , there is no chance to replace it with SSD ,

regarding Indexing , it is index per day , and there are indexes which are huge - 30g

I added additiional monitoring screens

[Christian_Dahlqvist]

If you are using networtked storage I would recommend you look at disk I/O, utilization and iowait, e.g. using iostat. It is quite likely this is what is limiting your performance so it is worth investigating this first.

[Burga]

Thank you ,

and the fact that on node4 there are less half number of shards ?

Best Regards.

[Christian_Dahlqvist]

If you have a number of reasonably sized shards, that means the average size of the others is even smaller which is not good. I stand by my recommendation to dramatically reduce the number of shards.

If node4 is the one you have highlighted it seems like it has an unusual number of very small shards. As I do not know how your indices are queried I can not tell whether this would lead to higher load than the others.

This is the node I would start looking at disk I/O on though.

[Burga]

Thank you .