Have 5 nodes cluster(each node has 16GB RAM out of which 8GB allocated to Elastic); The cluster has round 80% of analytics data which will be used for various heavy Kibana dashboard (accessed in less frequency) and remaining 20% of the data (separate indexes for live applications) is being manipulated by live applications.
Installed kibana in only one node - node5 and it is connected to the localnode (node-5).
We want to keep the cluster Highly Available for live applications, sometimes the kibana dashboard is occupying most of the cluster resources during heavy search and ideally impacting the cluster health and it is getting timeout for live applications; in 5 to 10 mins cluster is getting back to green (self-resilient) once all the pending tasks (kibana dashbord tasks) are completed. However, live applications will have a downtime for 5-10 mins during these times.
During these period CPU is ticking to 99% in 1 or 2 nodes and ideally whole cluster response is getting slow.
Should I split the cluster separately for analytics data and keep an another one for live applications? or should we need to evaluate and work on optimization and add additional hardware resources (like adding nodes, adding RAM..cpu etc) by maintaining in single cluster?
Which is the good practice in elastic perspective? Please suggest.
Thanks.
