# ElasticSearch Performance tuning with 3 nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [May 26, 2020, 3:11am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229 "2020-05-26T03:11:38Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 3:11am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/1 "2020-05-26T03:11:38Z")
Hi below is the configuration of my elasticstack. total of 3 nodes with all the nodes eligible for master as well as master node. each node system configuration RAM :- 8GB on each node CORE:- 4 core i have provided 4 GB to heap size to each of node. above are the configuration of my elasticstack. Now with the problem. I am doing load testing on elasticquery with 1000 concurrent user on 1 second. below is the query which i am using . ```auto { "query": { "bool": { "must": [ { "match_phrase": { "userLogin": { "query": "XXXX", "slop": 0, "zero_terms_query": "NONE", "boost": 1 } } }, { "match_phrase": { "targetSystemId": { "query": "3000", "slop": 0, "zero_terms_query": "NONE", "boost": 1 } } } ], "adjust_pure_negative": true, "boost": 1 } } } ``` **Above query will always return single unique result** Now with the result, thoughput is **252.44 per second with error of 1.15% connection timeout** As i think even if we use elasticsearch default configuration throughput of 252.44 is quite very low, i need to make it to atleast 1000 per second. please suggest how the above can be done. ---
**Author:** ![Brooke384](https://avatars.discourse-cdn.com/v4/letter/b/e9a140/32.png) [@Brooke384](https://discuss.elastic.co/u/Brooke384)\ **Post date:** [May 26, 2020, 3:37am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/2 "2020-05-26T03:37:31Z")
See here[:](https://www.quickpayportal.website/) [https://medium.com/kariyertech/elasticsearch-cluster-sizing-and-performance-tuning-42c7dd54de3c](https://medium.com/kariyertech/elasticsearch-cluster-sizing-and-performance-tuning-42c7dd54de3c) ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 4:59am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/3 "2020-05-26T04:59:25Z")
Thanks I will look into this, What I am thinking to troubleshoot this issue start with single index with default configuration and do the load testing. Then increase the shard, index and nodes as per the requirement Is it right approach to troubleshoot the issue ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 5:07am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/4 "2020-05-26T05:07:20Z")
I would recommend storing the two pieces of information you are filtering on in separate keyword mapped fields and then use term queries instead if match phrase. In order to give additional suggestions it would be good to know how many shards your data is distributed across and how much space this takes up on disk. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 7:29am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/5 "2020-05-26T07:29:25Z")
Thanks for the reply, below is the alternate query i already tried but there is not much difference in performance. ```auto { "query": { "bool": { "filter": { "bool": { "must": [ { "term": { "userLogin": "XXXXXXXX" } }, { "term":{ "targetSystemId": "3000" } } ] } } } } } ``` i have total of 3 nodes with 7 index with default configuration of 1 primary shard and 1 replica shard. below is the size taken by each index on primary as well as replica shard 1. first entity \*\*\*\*\*\*\*\*\*\*\* 47kb \*\*\*\*\*\*\*\*\*\*\*\*first and third node each 2. second entity \*\*\*\*\*\*\*\* 5mb \*\*\*\*\*\*\*\*\*\*\*\*first and second node each 3. third entity \*\*\*\*\*\*\*\*\*\*130mb \*\*\*\*\*\*\*\*\*\* first and third node each 4. fourth entity \*\*\*\*\*\*\*\*\* 1mb \*\*\*\*\*\*\*\*\*\*\*\*first and second node each 5. fifth entity \*\*\*\*\*\*\*\*\*\*\*3.3mb \*\*\*\*\*\*\*\*\*\*\*first and second nod each 6. sixth entity \*\*\*\*\*\*\*\*\*\*36.2kb \*\*\*\*\*\*\*\*\*\*\*third and first node each 7. seventh entity\*\*\*\*\*\*\*\*1mb \*\*\*\*\*\*\*\*\*\*\*\*\*second and first node each please let me know how i can improve the performance. Thanks a lot ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 9:45am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/6 "2020-05-26T09:45:24Z")
So you have 7 very small indices and a total of 14 shards? Why have you gone for having 7 indices instead of a single one? As long as you do not have any mapping conflicts I would recommend that you reindex all your data into a single index and set the number of replicas so that all data nodes hold a copy of the data. Then send queries distributed across all data nodes with a local preference. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 9:51am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/7 "2020-05-26T09:51:02Z")
all the 7 indices have different purpose on application level so basically i cannot merge those indices into one. so please suggest if anything else can be done. one more thing when doing load testing with apache jmeter of 7000 concurrent users i am getting below error. ```auto at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator.connect(HTTPHC4Impl.java:326) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:850) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:561) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:67) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1282) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1271) at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:627) at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:551) at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:490) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection timed out: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 19 more ``` **10.72.21.40 is data node currently this is first node, currently second node is master node** ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 10:04am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/8 "2020-05-26T10:04:35Z")
Start with a low concurrency level and gradually increase as long as query latency is acceptable. That will give you an idea of the level of concurrent queries your cluster can handle. If you can not consolidate your indices, which would make querying far more efficient, you may need more CPU cores to be able to handle more load in parallel. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/9 "2020-05-26T10:10:25Z")
Thanks for the prompt reply, Regarding concurrent queries cluster can handle is around 5500 concurrent users. for the more CPU core its already 4 core and **if utilization is not spiked is there is any need of more core ?.** currently CPU utilization is not more than 45% that means CPU is still not utilized on its full potential, right ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 10:48am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/10 "2020-05-26T10:48:10Z")
If CPU is not the bottleneck, try to find out what is. Given your low data volume it should not be disk I/O but could perhaps be networking. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 10:51am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/11 "2020-05-26T10:51:33Z")
thanks a lot, i will look into network side, but what about the error, its 30%, can it be due to networking ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 11:00am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/12 "2020-05-26T11:00:44Z")
Are you sending requests to all nodes in parallel? ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 11:03am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/13 "2020-05-26T11:03:38Z")
I am sending all the request to single data node, asi know elastic distributes the load across the cluster right ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 11:06am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/14 "2020-05-26T11:06:49Z")
You should distribute it across all data nodes. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 11:31am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/15 "2020-05-26T11:31:07Z")
can you let me know, how to do the same. it will be greatly helpful. Thanks a lot ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 26, 2020, 11:32am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/16 "2020-05-26T11:32:01Z")
That is something you need to set up in JMeter. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 11:34am UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/17 "2020-05-26T11:34:33Z")
ok, thanks a lot i will do the same. ---
**Author:** ![Roshan\_Jha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roshan_jha/32/50716_2.png) [@Roshan\_Jha](https://discuss.elastic.co/u/Roshan_Jha)\ **Post date:** [May 26, 2020, 12:12pm UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/18 "2020-05-26T12:12:11Z")
Hi Christian, In log i see something like this **{ml.machine\_memory=8191995904, ml.max\_open\_jobs=20, xpack.installed=true}** is this a matter of concern and **is it possible to disable ml since i am not using it.** ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 23, 2020, 12:12pm UTC](https://discuss.elastic.co/t/elasticsearch-performance-tuning-with-3-nodes/234229/19 "2020-06-23T12:12:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.