# Adding nodes does not seem to speed up indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [May 25, 2010, 7:35pm UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972 "2010-05-25T19:35:10Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [May 25, 2010, 7:35pm UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/1 "2010-05-25T19:35:10Z")
I ran load tests to find out how fast I could index documents. I am using the java client with ElasticSearch 0.7.0, and I found that adding more data nodes didn't really speed things up. Having only one data node was faster; I guess because it did not have to do any routing or balancing, but it is not sustainable for my needs. Having 2 and 4 data nodes gave me about the same transactions per second. Is this expected? Any advice on speeding up indexing throughput? Thanks. Threads Itr/thread Data Nodes Trans per Sec 20 2000 1 1884.39 20 2000 2 946.10 20 2000 4 1061.04 40 4000 4 876.77 ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 26, 2010, 6:23am UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/2 "2010-05-26T06:23:15Z")
Are you maxing up on resources no your client side? Also, if you use the default 5 shards each with 1 replica, with only one node, there will only be 5 shards allocated. With 2 nodes, 10 shards will be allocated, and index operation will need to be replicated to another node. Which client do you use? TransportClient or Node client? cheers, shay.banon On Tue, May 25, 2010 at 10:35 PM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > I ran load tests to find out how fast I could index documents. I am using > the java client with Elasticsearch 0.7.0, and I found that adding more data > nodes didn't really speed things up. Having only one data node was faster; > I guess because it did not have to do any routing or balancing, but it is > not sustainable for my needs. Having 2 and 4 data nodes gave me about the > same transactions per second. Is this expected? Any advice on speeding > up > indexing throughput? Thanks. > > ## Threads Itr/thread Data Nodes Trans per Sec 20 2000 1 1884.39 20 2000 2 946.10 20 2000 4 1061.04 40 4000 4 876.77 > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Adding-nodes-does-not-seem-to-speed-up-indexing-tp842987p842987.html](http://elasticsearch-users.115913.n3.nabble.com/Adding-nodes-does-not-seem-to-speed-up-indexing-tp842987p842987.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 26, 2010, 7:15am UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/3 "2010-05-26T07:15:41Z")
Also, how are you indexing the data? Is it a single thread indexing? In this case, you will not see improvement. On Wed, May 26, 2010 at 9:23 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Are you maxing up on resources no your client side? Also, if you use the > default 5 shards each with 1 replica, with only one node, there will only be > 5 shards allocated. With 2 nodes, 10 shards will be allocated, and index > operation will need to be replicated to another node. > > Which client do you use? TransportClient or Node client? > > cheers, > shay.banon > > On Tue, May 25, 2010 at 10:35 PM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > > > I ran load tests to find out how fast I could index documents. I am using > > the java client with Elasticsearch 0.7.0, and I found that adding more > > data > > nodes didn't really speed things up. Having only one data node was > > faster; > > I guess because it did not have to do any routing or balancing, but it is > > not sustainable for my needs. Having 2 and 4 data nodes gave me about the > > same transactions per second. Is this expected? Any advice on speeding > > up > > indexing throughput? Thanks. > > > > ## Threads Itr/thread Data Nodes Trans per Sec 20 2000 1 1884.39 20 2000 2 946.10 20 2000 4 1061.04 40 4000 4 876.77 > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Adding-nodes-does-not-seem-to-speed-up-indexing-tp842987p842987.html](http://elasticsearch-users.115913.n3.nabble.com/Adding-nodes-does-not-seem-to-speed-up-indexing-tp842987p842987.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 26, 2010, 11:28am UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/4 "2010-05-26T11:28:34Z")
On Tue, 2010-05-25 at 12:35 -0700, John Chang wrote: > I ran load tests to find out how fast I could index documents. I am using > the java client with Elasticsearch 0.7.0, and I found that adding more data > nodes didn't really speed things up. Having only one data node was faster; > I guess because it did not have to do any routing or balancing, but it is > not sustainable for my needs. Having 2 and 4 data nodes gave me about the > same transactions per second. Is this expected? Any advice on speeding up > indexing throughput? Thanks. I've seen the same thing, and kimchy has just explained to me on IRC how things should work, so I thought I'd share it here for those who, like me, are new to distributed workloads. There are two concerns being addresses in a cluster:  - performance/scalability - high availability/redundancy There are three variables which we can change to affect the above:  - index.number\_of\_shards (default 5) - index.number\_of\_replicas (default 2) - the number of nodes we start index.number\_of\_shards is intended to improve performance - the more shards (within reason) the more we can divide up the work between nodes. index.number\_of\_replicas is intended to improve redundancy AND search performance. - The more replicas, the better chance that our cluster will keep going when a node goes down. The number of replicas is in addition to the primary shard, so with the default of 2, we have the primary plus 2 replicas (a total of 3) - The more replicas, the more that search requests can be distributed across more nodes Having more replicas comes at a cost. A newly indexed document needs to be added to the primary shard, then copied to each replica. So for example: - we start 3 nodes - we create an index with 5 shards and 2 replicas. - this gives us a total of 15 shards (5 primaries and 10 replicas). - searching across this cluster is faster than searching against one node, as the data is spread across 3 nodes - indexing a new doc in this cluster will be slightly slower than indexing to a single node, because of the replication costs Now we add a fourth node. - we still have 15 shards, but now they are spread across 4 machines, - searching speed improves because the data is spread across more machines - indexing speed improves because now each node is doing less work than a single node So to see similar indexing performance improvements, you could decrease the number of replicas to 1, but then you're potentially increasing the risk of cluster downtime. This is as I understand it - please correct me if I have something wrong. clint -- Web Announcements Limited is a company registered in England and Wales, with company number 05608868, with registered address at 10 Arvon Road, London, N5 1PR. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 26, 2010, 11:39am UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/5 "2010-05-26T11:39:37Z")
Basically its correct. A few notes: 1. The default number of replicas is 1 and not 2 :). 2. With more replicas, it means that each shard that exists within the replication group is basically doing the indexing. Note, replication is done using async IO, so its not a thread per shard target waiting for it to finish. Also, I plan to add async replication, which will improve performance by paying in being async. On Wed, May 26, 2010 at 2:28 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > On Tue, 2010-05-25 at 12:35 -0700, John Chang wrote: > > > I ran load tests to find out how fast I could index documents. I am > > using > > the java client with Elasticsearch 0.7.0, and I found that adding more > > data > > nodes didn't really speed things up. Having only one data node was > > faster; > > I guess because it did not have to do any routing or balancing, but it is > > not sustainable for my needs. Having 2 and 4 data nodes gave me about > > the > > same transactions per second. Is this expected? Any advice on speeding > > up > > indexing throughput? Thanks. > > I've seen the same thing, and kimchy has just explained to me on IRC how > things should work, so I thought I'd share it here for those who, like > me, are new to distributed workloads. > > There are two concerns being addresses in a cluster: >  - performance/scalability > > - high availability/redundancy > > There are three variables which we can change to affect the above: >  - index.number\_of\_shards (default 5) > > - index.number\_of\_replicas (default 2) > - the number of nodes we start > > index.number\_of\_shards is intended to improve performance - the more > shards (within reason) the more we can divide up the work between nodes. > > index.number\_of\_replicas is intended to improve redundancy AND search > performance. > > - The more replicas, the better chance that our cluster will keep going > when a node goes down. The number of replicas is in addition to the > primary shard, so with the default of 2, we have the primary plus 2 > replicas (a total of 3) > - The more replicas, the more that search requests can be distributed > across more nodes > > Having more replicas comes at a cost. A newly indexed document needs to > be added to the primary shard, then copied to each replica. > > So for example: > > - we start 3 nodes > - we create an index with 5 shards and 2 replicas. > - this gives us a total of 15 shards (5 primaries and 10 replicas). > - searching across this cluster is faster than searching against one > node, as the data is spread across 3 nodes > - indexing a new doc in this cluster will be slightly slower than > indexing to a single node, because of the replication costs > > Now we add a fourth node. > > - we still have 15 shards, but now they are spread across 4 machines, > - searching speed improves because the data is spread across more > machines > - indexing speed improves because now each node is doing less work > than a single node > > So to see similar indexing performance improvements, you could decrease > the number of replicas to 1, but then you're potentially increasing the > risk of cluster downtime. > > This is as I understand it - please correct me if I have something > wrong. > > clint > > -- > Web Announcements Limited is a company registered in England and Wales, > with company number 05608868, with registered address at 10 Arvon Road, > London, N5 1PR. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 26, 2010, 11:49am UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/6 "2010-05-26T11:49:10Z")
One more thing, you should do much more iterations to get proper performance numbers. And discard the first 10,000 because of "warming" of the servers. On Wed, May 26, 2010 at 2:39 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Basically its correct. A few notes: > > 1. The default number of replicas is 1 and not 2 :). > 2. With more replicas, it means that each shard that exists within the > replication group is basically doing the indexing. Note, replication is done > using async IO, so its not a thread per shard target waiting for it to > finish. Also, I plan to add async replication, which will improve > performance by paying in being async. > > On Wed, May 26, 2010 at 2:28 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > > > On Tue, 2010-05-25 at 12:35 -0700, John Chang wrote: > > > > > I ran load tests to find out how fast I could index documents. I am > > > using > > > the java client with Elasticsearch 0.7.0, and I found that adding more > > > data > > > nodes didn't really speed things up. Having only one data node was > > > faster; > > > I guess because it did not have to do any routing or balancing, but it > > > is > > > not sustainable for my needs. Having 2 and 4 data nodes gave me about > > > the > > > same transactions per second. Is this expected? Any advice on > > > speeding up > > > indexing throughput? Thanks. > > > > I've seen the same thing, and kimchy has just explained to me on IRC how > > things should work, so I thought I'd share it here for those who, like > > me, are new to distributed workloads. > > > > There are two concerns being addresses in a cluster: > >  - performance/scalability > > > > - high availability/redundancy > > > > There are three variables which we can change to affect the above: > >  - index.number\_of\_shards (default 5) > > > > - index.number\_of\_replicas (default 2) > > - the number of nodes we start > > > > index.number\_of\_shards is intended to improve performance - the more > > shards (within reason) the more we can divide up the work between nodes. > > > > index.number\_of\_replicas is intended to improve redundancy AND search > > performance. > > > > - The more replicas, the better chance that our cluster will keep going > > when a node goes down. The number of replicas is in addition to the > > primary shard, so with the default of 2, we have the primary plus 2 > > replicas (a total of 3) > > - The more replicas, the more that search requests can be distributed > > across more nodes > > > > Having more replicas comes at a cost. A newly indexed document needs to > > be added to the primary shard, then copied to each replica. > > > > So for example: > > > > - we start 3 nodes > > - we create an index with 5 shards and 2 replicas. > > - this gives us a total of 15 shards (5 primaries and 10 replicas). > > - searching across this cluster is faster than searching against one > > node, as the data is spread across 3 nodes > > - indexing a new doc in this cluster will be slightly slower than > > indexing to a single node, because of the replication costs > > > > Now we add a fourth node. > > > > - we still have 15 shards, but now they are spread across 4 machines, > > - searching speed improves because the data is spread across more > > machines > > - indexing speed improves because now each node is doing less work > > than a single node > > > > So to see similar indexing performance improvements, you could decrease > > the number of replicas to 1, but then you're potentially increasing the > > risk of cluster downtime. > > > > This is as I understand it - please correct me if I have something > > wrong. > > > > clint > > > > -- > > Web Announcements Limited is a company registered in England and Wales, > > with company number 05608868, with registered address at 10 Arvon Road, > > London, N5 1PR. ---
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [May 26, 2010, 6:50pm UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/7 "2010-05-26T18:50:20Z")
Thanks for the info. I'll try again, ignoring the first 10k. To answer your questions above, I am using the java-based node clientfrom the 0.7.1 release, and my client is multi-threaded. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 28, 2010, 10:20pm UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/8 "2010-05-28T22:20:32Z")
Yea, don't know how I missed the multi threaded part in my second response, too late I guess... . I would also check if your stress client is not maxing up on resources (CPU, Mem). One more thing, is that on a single index, or several indices? On Wed, May 26, 2010 at 9:50 PM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > ## Thanks for the info. I'll try again, ignoring the first 10k. To answer your questions above, I am using the java-based node clientfrom the 0.7.1 release, and my client is multi-threaded. > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Adding-nodes-does-not-seem-to-speed-up-indexing-tp842987p845702.html](http://elasticsearch-users.115913.n3.nabble.com/Adding-nodes-does-not-seem-to-speed-up-indexing-tp842987p845702.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:23am UTC](https://discuss.elastic.co/t/adding-nodes-does-not-seem-to-speed-up-indexing/2972/9 "2017-07-06T04:23:43Z")