# Elastic and Kibana Load Balancing **URL:** \ **Category:** Elasticsearch\ **Created:** [July 12, 2019, 1:32pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251 "2019-07-12T13:32:15Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 12, 2019, 1:32pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/1 "2019-07-12T13:32:16Z")
I have 2 physical machines where I run Elastic and Kibana on 120 - Master Node and a Data Node 121 - Data Node I have 8C/64GB Ram and 4TB of hard drive space on them. the clusters are setup and the indices have replicated on both 120 and 121. My issues is on Kibana. Should kibana run on 120 and 121? Or just 120 or 121? Also I get the 30000ms timeout errors because I have approximately 3 billion transactions in Elastic that people run searches/dashboards on. And quite a bit of times, especially if 2 or more users are doing something, the system times out. How do I solve this issue? ---
**Author:** ![jonathan\_rowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_rowe/32/49275_2.png) [@jonathan\_rowe](https://discuss.elastic.co/u/jonathan_rowe)\ **Post date:** [July 12, 2019, 2:30pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/2 "2019-07-12T14:30:41Z")
I'd run it on 121, as you have the master using memory on 120 What size is your heap? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 12, 2019, 3:32pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/3 "2019-07-12T15:32:37Z")
Heap is set at 28GB. Also note that data ONLY gets added to Elastic once a day! About 10 million records inserted. ---
**Author:** ![jonathan\_rowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathan_rowe/32/49275_2.png) [@jonathan\_rowe](https://discuss.elastic.co/u/jonathan_rowe)\ **Post date:** [July 13, 2019, 9:12am UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/4 "2019-07-13T09:12:49Z")
So you're a mainly RO use case, merge your segments to a reasonable degree post load - merging to a single segment will probably take too long with 64gb RAM and a HDD You could increase the search timeout (if acceptable) If not, then you need to either optimise your mappings/indices or add more hardware ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 11:15am UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/5 "2019-07-13T11:15:12Z")
How large are your indices? How many indices and shards do you have in the cluster? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 11:21am UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/6 "2019-07-13T11:21:19Z")
What's an RO? and I do have SSDs sorry for not clarifying earlier. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 11:23am UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/7 "2019-07-13T11:23:26Z")
My seed index is about 800GB. It's the largest one. Has 1.1 billion transactions. The rest are around 10 million each per day and are about 10gb in size. At the moment I only have 2 total indices. Shards setting is at default. So not entirely sure. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 11:44am UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/8 "2019-07-13T11:44:32Z")
Can you provide the output of the `_cat/indices` API? It sounds like you have quite large shards, which can impact query performance. What type of queries are you running? What does CPU usage, disk I/O and iowait look like while you are querying? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 2:53pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/9 "2019-07-13T14:53:26Z")
> [@Christian\_Dahlqvist](#): > > \_cat/indices ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24b9ade975ae5b0631cf9963e3ec130ed03e2a67.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 13, 2019, 3:18pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/10 "2019-07-13T15:18:04Z")
Please don't post images of text as they are hardly readable and not searchable. Instead paste the text and format it with `` icon. Check the preview window. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 3:26pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/11 "2019-07-13T15:26:16Z")
I can see that you have one index with a single primary shard of 894.1GB. The replica is the same size. This is likely going to be very slow to query as each query is single-threaded against each shard. I would recommend you reindex this into a new index with e.g. 30 primary shards. When you do so I would recommend you also set the `number_of_routing_shards` to e.g. 120 so you later on can use the [split index API](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-split-index.html) if needed. This assume you are on a reasonablt recent version. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 3:31pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/12 "2019-07-13T15:31:03Z")
Will post in a minute, just to play with some settings I am trying to reindex and change the default to 5 shards and 2 replicas. I have 2 physical nodes. I want to see if the performance changes if the index is sharded. I understand that 1TB is a fairly large and has to be reindexed. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 3:31pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/13 "2019-07-13T15:31:15Z")
We are on the latest and greatest 7.2! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 3:32pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/14 "2019-07-13T15:32:10Z")
Then you MAY be able to use the split index API without reindexing. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 3:32pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/15 "2019-07-13T15:32:11Z")
Wow 30 primaries? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 3:32pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/16 "2019-07-13T15:32:55Z")
Yes, that will get you down to 30GB per shard. You could go a bit bigger, but that will allow you to grow for a while. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 3:33pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/17 "2019-07-13T15:33:35Z")
So you would just change "number\_of\_shards" : 50 and "number\_of\_replicas" : 2 (is 2 enough?) ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 13, 2019, 3:34pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/18 "2019-07-13T15:34:20Z")
we split daily indexes into smaller bits and they are generally 10GB per day so 30 or 50 shards would mean 500MB indexes, would that be an issue? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 3:34pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/19 "2019-07-13T15:34:43Z")
First check what the index settings of the large index is. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 13, 2019, 3:36pm UTC](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251/20 "2019-07-13T15:36:11Z")
The split is for the really large index only. You want to avoid having lots of really small indices so try to keep the shard size above 10GB for time-based indices. [Next page](https://discuss.elastic.co/t/elastic-and-kibana-load-balancing/190251.md?page=2)