Making Kibana (4) searches faster

Elastic Stack Elasticsearch
1

Hi,

Kindly help me with some tips on tuning elasticseach for faster searching. Sorry for the long post.

I have two elasticsearch clusters

Two bare metal nodes, each with 24 cores, 64 GiB RAM, rotational 1TB HDDAround 5853319 docs (10GiB) per day with 4 primary shards and replication=116GiB heap, no other special configuration

In this cluster, if I do a 7 day search in Kibana 4 dashboard with
around 30 graphs, it takes more than a minute to return all results,
then on applying a filter, it takes another minute to give back results.
Can it do it faster than this?

HQ plugin shows the following stats in warning/error stages for the two nodes:

Search - Query: 308.35ms 355.12ms

Search - Fetch: 104.61ms 84.6ms

Refresh: 34.63ms 29.37ms

Three EC2 c4.4xlarge data nodes, each with 16 cores, 30 GiB RAM, EBS volumesAround 188361074 docs (400GiB) per day with 5 primary shards and replication=116GiB heap

Was getting error in Kibana, so increased circuitbreaker to 75%. Then
did a 7 day search, brought the whole cluster down due to OOM
exceptions. Was that overkill? What can I do to make this search
successful, and to prevent the cluster from going down?

Hoping for some pointers here. Thanks

2

Check your slow logs and and hot threads, because KB will only be as fast as ES.

The first cluster is probably being held back by the disk.
The second one may just be overloaded, can you add more nodes or close some older indices and try again? Increasing the field data circuit breaker like that is dangerous, look into doc values to help.