I have used the Dockerization concepts for my ELK implementation. Please find my ELK node infrastructure as below,
I have created around 15 dashboards. Each dashboard has around 10 to 15 visualizations. And loading more than 10K+ records in some visualization. Now, while accessing those dashboards am facing some performance issue (sometimes timeout issue occurring). When I select the more than a 1-day range in kibana it's totally down.
Could someone please help me out to fix this issue.
Totally I have 2332 shards, 1074 Indices, 492,080,781 documents and am using X-Pack trial pack so monitored the node performance through kibana. While accessing the dashboard the elastic Search Latency is 24.8/s, Index Latency is: 0.23ms and Total shards Search Rate is 28.2/s. And that dashboard took almost 3 mins to load completely.
Also, I have checked the individual docker container CPU, Mem usage as below,
Given that you only have two nodes in your cluster, your shard count sounds quite high, which could very well be contributing to your problems. Please read this blog post for some practical guidance on shards and sharding.
It would also be interesting to know what types of visualisations and aggregations you have in your dashboards. You mentioned loading 10k documents in some dashboards. Is this saved searches embedded in the dashboards?
I will try to add one more node to handle my shards and at the same time try to reduce the shards. And am not using the saved searches. Mostly I have used the Bar, Data table visualization. Coming to aggregation part Terms, Filters aggregation used in Buckets and TopHits aggregation in Metric.
Here am attaching one of the sample Bar charts. In x-axis, I have some terms and each column trying to showcase the more than 1000 counts in the stacked view. The same way is loading more than 10K data in the data table.