# Elastic search cluster setup suggestion required **URL:** \ **Category:** Elasticsearch\ **Created:** [September 20, 2018, 9:13am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258 "2018-09-20T09:13:16Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\ **Post date:** [September 20, 2018, 9:13am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/1 "2018-09-20T09:13:17Z")
Hi, I have a cluster setup of 2 data nodes and 1 master node. One eligible master. I handle about 400Gb of data. All our search requests and indexing requests are made to the master node. This caused some RAM issues on master while searching. **Need suggestion** 1. Should we have one coordinating node and route all requests to this node ? 2. Is having one coordinating node a better choice or 2 ? 2.1 To avoid split brain we want to make 3 eligible masters. Which ones ? 3. Is it a good idea to make Data node an eligible master ? 4. Is it a good idea to make search requests to the data nodes ? 5. While initializing RestSearchClient, which hostnames should we mention for search ? 5.1 My guess was 2 coordinating node hostnames ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2018, 9:21am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/2 "2018-09-20T09:21:05Z")
> [@pokaleshrey](#): > > I have a cluster setup of 2 data nodes and 1 master node. One eligible master. In any cluster you generally want to have 3 master-eligible nodes (with `minimum_master_nodes` set to `2` to avoid split brain scenarios). This will allow the cluster to stay responsive even if it loses one node. > [@pokaleshrey](#): > > All our search requests and indexing requests are made to the master node. If you have a dedicated master node, which generally is less powerful that other nodes, which should be left to just managing the cluster and not serve traffic. > [@pokaleshrey](#): > > Should we have one coordinating node and route all requests to this node ? Whether coordinating-only node help or not depends on the use-case and workload. > [@pokaleshrey](#): > > Is having one coordinating node a better choice or 2 ? If you have a single coordinating-only node that you send all requests to, this becomes a single point of failure, so 2 is generally better (if you need them at all). > [@pokaleshrey](#): > > Is it a good idea to make Data node an eligible master ? It is important to have 3 master-eligible nodes, so unless you have 3 dedicated master node this is generally a good idea. > [@pokaleshrey](#): > > Is it a good idea to make search requests to the data nodes ? Often this is fine, but it does as I mentioned earlier depend on the use-case. > [@pokaleshrey](#): > > While initializing RestSearchClient, which hostnames should we mention for search ? > 5.1 My guess was 2 coordinating node hostnames Coordinating-only nodes if you have them in the cluster, otherwise data nodes. ---
**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\ **Post date:** [September 20, 2018, 9:40am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/3 "2018-09-20T09:40:47Z")
Thanks @Christian_Dahlqvist . Appreciate the inputs. With your inputs we have a plan: 1 Master node 2 Coordinating nodes (eligible masters) 2 Data nodes (One Replica)(RAID-0) Avg doc size = 4Kb Total docs = 250million One Index. Currently optimal 6 shards for our usecase. We use routing while indexing and searching. **Few more Queries** 1. Can we increase search throughput by adding more data nodes and more replicas ? 2. Would this infrastructure cause impact on indexing throughput ? 3. Does this setup look good to you ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2018, 10:09am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/4 "2018-09-20T10:09:57Z")
> [@pokaleshrey](#): > > Can we increase search throughput by adding more data nodes and more replicas ? Yes, that is generally the way to increase search throughput. > [@pokaleshrey](#): > > Would this infrastructure cause impact on indexing throughput ? Not sure what you mean. Adding additional replicas will however have an impact on indexing load. > [@pokaleshrey](#): > > Does this setup look good to you ? Assuming you have 3 master-eligible nodes it sounds reasonable, but I do not know as I am not familiar with your data, query load or requirements. ---
**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\ **Post date:** [September 20, 2018, 10:21am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/5 "2018-09-20T10:21:31Z")
Thanks again. One last thing i was wondering about. In past we faced OOMemory from kernel which killed java process on master node and we had to allocate more RAM for OS to continue operations. This was when we were directing requests to master. **Query** : Why was my OS memory of master node being utilized, when the master is not really using LUCENE operations to cache segments. All my data operations are made on data node. Master node is just scattering and gathering docs ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2018, 10:23am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/6 "2018-09-20T10:23:05Z")
> [@pokaleshrey](#): > > **Query** : Why was my OS memory of master node being utilized, when the master is not really using LUCENE operations to cache segments. All my data operations are made on data node. Master node is just scattering and gathering docs ? Parsing and coordinating requests can consume a fair bit of heap as well. How much load do you serve and how much heap did you have assigned? ---
**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\ **Post date:** [September 20, 2018, 10:35am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/7 "2018-09-20T10:35:47Z")
Our Master node was 8core 32bit machine. Alloted heap space was recommended 50% i.e. 16Gb. The space available for OS was the remaining 16Gb. We were serving 3000 ES requests per second, thats about 50Mb of docs per sec. The kernel OOMemory killer killed our master node java process and cluster was down. _Note:_ Java was not out of heap. OS did not get memory and it killed java to free up space. **Our solution:** We have put _upgraded existing machine_ 16core and 64Gb. Allotted heap space now is 16Gb and OS gets remaining 48Gb to play. Now things are good. **Query Again:** Does master do any kind of LUCENE related operations ? AFAIK it just does QUERY and FETCH. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2018, 10:38am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/8 "2018-09-20T10:38:03Z")
If you are running on a machine with a 32-bit architecture I believe you must have been running with a much smaller heap. Or was that a typo? I would recommend installing monitoring (if you do not already have it) so you can see how heap usage varies over time. ---
**Author:** ![pokaleshrey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pokaleshrey/32/48680_2.png) [@pokaleshrey](https://discuss.elastic.co/u/pokaleshrey)\ **Post date:** [September 20, 2018, 10:40am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/9 "2018-09-20T10:40:34Z")
> [@Christian\_Dahlqvist](#): > > If you are running on a machine with a 32-bit architecture I believe you must have been running with a much smaller heap. Or was that a typo? OMG blunder. My Mistake. I wanted to say 32Gb there. We use all 64-bit archi machines. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 18, 2018, 10:40am UTC](https://discuss.elastic.co/t/elastic-search-cluster-setup-suggestion-required/149258/10 "2018-10-18T10:40:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.