# Will having multiple nodes in a cluster speed up indexing? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 27, 2013, 12:49am UTC](https://discuss.elastic.co/t/will-having-multiple-nodes-in-a-cluster-speed-up-indexing/12963 "2013-07-27T00:49:35Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Kiran\_Madabhushi](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@Kiran\_Madabhushi](https://discuss.elastic.co/u/Kiran_Madabhushi)\ **Post date:** [July 27, 2013, 12:49am UTC](https://discuss.elastic.co/t/will-having-multiple-nodes-in-a-cluster-speed-up-indexing/12963/1 "2013-07-27T00:49:35Z")
Hi I am using elasticsearch and logstash for managing logs from a very big system. Right now, during the test phase, I am using 1 machine. It indexes about 2000 logs messages per second. If I have 10 million log messages, it takes a few hours for the indexing to complete. (I dont know if its the indexing thats taking a long time or logstash for filtering the messages). If I a create a cluster of ES machines, will this speed up indexing? I am not really concerned about replicas. Can I configure ES nodes to do just the indexing part and not worry about replicas. Please suggest any techniques Thanks Kiran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [July 30, 2013, 4:16pm UTC](https://discuss.elastic.co/t/will-having-multiple-nodes-in-a-cluster-speed-up-indexing/12963/2 "2013-07-30T16:16:33Z")
Hello Kiran, Yes, you can change the number of replicas on the fly using the Update Settings API: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... So you can set the number of replicas to 0 and have only your primary shards balanced across your cluster. If this case, adding more nodes will help your indexing speed, as long as you have enough shards to spread on all your nodes. As Logstash uses daily indices by default, you'll probably want to make sure that shards of today's index (which are hit with indexing requests) are evenly distributed. A simple way of doing this is with the index.routing.allocation.total\_shards\_per\_node setting: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... For example, if you have 10 shards per index (no replicas) and 5 nodes, set that number to 2. There are quite a lot of tricks to get your indexing speed up. Although, there's almost always a trade-off. Here are the top 3 (IMO): - use the bulk API [http://www.elasticsearch.org/guide/reference/api/bulk/](http://www.elasticsearch.org/guide/reference/api/bulk/). The trade-off being you'll have to use the elasticsearch\_http output [http://logstash.net/docs/1.1.13/outputs/elasticsearch\_http](http://logstash.net/docs/1.1.13/outputs/elasticsearch_http)for that, at the moment - increase the refresh interval. Here's a blog post[http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/](http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/)about it. The trade-off is that your searches will be less up-to-date - increase the indexing buffer size[http://www.elasticsearch.org/guide/reference/modules/indices/](http://www.elasticsearch.org/guide/reference/modules/indices/). The trade-off is you'll have less memory for stuff like searches You may want to monitor your cluster to check out what work and what doesn't, and where are your bottlenecks. If you're looking for a monitoring tool for Elasticsearch, check out our SPM: > **[Elasticsearch - Sematext Documentation](https://sematext.com/docs/integration/elasticsearch-integration/)** > > Collect and monitor key Elasticsearch metrics such as request latency, indexing rate, and segment merges with built-in anomaly detection, threshold, and heartbeat alerts. Send notifications to email and various chatops messaging services, correlate... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Sat, Jul 27, 2013 at 3:49 AM, Kiran Madabhushi [maskiran@gmail.com](mailto:maskiran@gmail.com)wrote: > Hi > I am using elasticsearch and logstash for managing logs from a very big > system. Right now, during the test phase, I am using 1 machine. It indexes > about 2000 logs messages per second. If I have 10 million log messages, it > takes a few hours for the indexing to complete. (I dont know if its the > indexing thats taking a long time or logstash for filtering the messages). > If I a create a cluster of ES machines, will this speed up indexing? I am > not really concerned about replicas. Can I configure ES nodes to do just > the indexing part and not worry about replicas. Please suggest any > techniques > > Thanks > Kiran > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:23am UTC](https://discuss.elastic.co/t/will-having-multiple-nodes-in-a-cluster-speed-up-indexing/12963/3 "2017-07-06T02:23:43Z")