# Increasing shards and then nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [January 19, 2011, 5:29pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810 "2011-01-19T17:29:42Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Lee\_Parker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lee_parker/32/3250_2.png) [@Lee\_Parker](https://discuss.elastic.co/u/Lee_Parker)\ **Post date:** [January 19, 2011, 5:29pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/1 "2011-01-19T17:29:42Z")
We have been using elastic search within our production environment for a few months now. I have discovered that I didn't properly plan for the amount of data in the index and find myself in need of increasing the number of shards to improve performance. I am also going to increase the number of nodes. I know that you can't just increase the number of shards without reindexing the data. The difficulty in reindexing the data for me is that our data currently resides in a highly normalized system and de-normalizing each item into the document that ES indexes can take a long time (2.5 weeks last time). In that case, here is my plan for increasing my shard count: 1. Spin up a new ES server but make sure it doesn't join the existing server as a node. 2. Pull the list of documents from my data and then get the documents from the existing ES server and put them on the new one. 3. Once all the data is in the new ES server, shutdown the older one wipe its data and then start it up as a new node to join the new ES server. Is there a better way to keep things going and increase the shard count? We are currently using the 5 shard default and all of the shards are around 12G. I am thinking about setting it to 15 shards and 1 (possibly 2) replicas. Does this make sense? We are going to have a 3 node cluster by the end of the month and will likely increase that as we see fit throughout the year. ## Lee "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." by Marty Beckerman ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [January 19, 2011, 5:39pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/2 "2011-01-19T17:39:21Z")
Hi Lee > In that case, here is my plan for increasing my shard count: > > 1. Spin up a new ES server but make sure it doesn't join the existing > server as a node. > 2. Pull the list of documents from my data and then get the documents > from the existing ES server and put them on the new one. > 3. Once all the data is in the new ES server, shutdown the older one > wipe its data and then start it up as a new node to join the new ES > server. Alternatively, instead of messing with a new server, you could: - create a new index 'new\_index\_timestamp' on the same ES server - index from 'my\_index' to 'new\_index\_timestamp' - delete 'my\_index' - create alias 'my\_index' pointing to 'new\_index\_timestamp' The benefit of switching to using aliases is that it will make it easier to make changes to your index in the future. clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 7:22pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/3 "2011-01-19T19:22:46Z")
The 5 shards default allow you to grow up to 5 nodes in terms of index capacity, and in terms of improving search capacity, you can simply dynamically add more replicas to the cluster. So, with a 5 shard 1 replica option, you will max out on 10 nodes (nothing will be left to be allocated to the 11th node if started). Sounds like you are planning on growing from 1-2 nodes to 3, this is still well below the capacity provided by 5 shards and 1 replica. A 12gb index is not a big one. -shay.banon On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > Hi Lee > > > In that case, here is my plan for increasing my shard count: > > > > 1. Spin up a new ES server but make sure it doesn't join the existing > > server as a node. > > 2. Pull the list of documents from my data and then get the documents > > from the existing ES server and put them on the new one. > > 3. Once all the data is in the new ES server, shutdown the older one > > wipe its data and then start it up as a new node to join the new ES > > server. > > Alternatively, instead of messing with a new server, you could: > > - create a new index 'new\_index\_timestamp' on the same ES server > - index from 'my\_index' to 'new\_index\_timestamp' > - delete 'my\_index' > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > The benefit of switching to using aliases is that it will make it easier > to make changes to your index in the future. > > clint ---
**Author:** ![Lee\_Parker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lee_parker/32/3250_2.png) [@Lee\_Parker](https://discuss.elastic.co/u/Lee_Parker)\ **Post date:** [January 20, 2011, 2:01pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/4 "2011-01-20T14:01:04Z")
It was my understanding based on some other threads that performance will degrade once shards are greater than 10g. If that is not true then I will just stick with the 5 shards and 1 replica setup. ## Lee "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." by Marty Beckerman On Wed, Jan 19, 2011 at 1:22 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > The 5 shards default allow you to grow up to 5 nodes in terms of index > capacity, and in terms of improving search capacity, you can simply > dynamically add more replicas to the cluster. So, with a 5 shard 1 replica > option, you will max out on 10 nodes (nothing will be left to be allocated > to the 11th node if started). > > Sounds like you are planning on growing from 1-2 nodes to 3, this is still > well below the capacity provided by 5 shards and 1 replica. A 12gb index is > not a big one. > > -shay.banon > > On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > > Hi Lee > > In that case, here is my plan for increasing my shard count: > > 1. Spin up a new ES server but make sure it doesn't join the existing > server as a node. > 2. Pull the list of documents from my data and then get the documents > from the existing ES server and put them on the new one. > 3. Once all the data is in the new ES server, shutdown the older one > wipe its data and then start it up as a new node to join the new ES > server. > > Alternatively, instead of messing with a new server, you could: > > - create a new index 'new\_index\_timestamp' on the same ES server > - index from 'my\_index' to 'new\_index\_timestamp' > - delete 'my\_index' > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > The benefit of switching to using aliases is that it will make it easier > to make changes to your index in the future. > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 20, 2011, 4:18pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/5 "2011-01-20T16:18:41Z")
It really depends what you plan to do with it. They can certainly be bigger than 10gb, if you use things like facets or sorting, the values of the field per doc will be loaded to memory, which means that you might need more memory as the shard doc count grows, but that is linear growth, that can be easily derived from some capacity planning down on a smaller size. -shay.banon On Thursday, January 20, 2011 at 4:01 PM, Lee Parker wrote: > It was my understanding based on some other threads that performance will degrade once shards are greater than 10g. If that is not true then I will just stick with the 5 shards and 1 replica setup. > > ## Lee > > "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." > by Marty Beckerman > > On Wed, Jan 19, 2011 at 1:22 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > The 5 shards default allow you to grow up to 5 nodes in terms of index capacity, and in terms of improving search capacity, you can simply dynamically add more replicas to the cluster. So, with a 5 shard 1 replica option, you will max out on 10 nodes (nothing will be left to be allocated to the 11th node if started). > > > > Sounds like you are planning on growing from 1-2 nodes to 3, this is still well below the capacity provided by 5 shards and 1 replica. A 12gb index is not a big one. > > > > -shay.banon > > On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > > > > > Hi Lee > > > > > > > In that case, here is my plan for increasing my shard count: > > > > > > > > 1. Spin up a new ES server but make sure it doesn't join the existing > > > > server as a node. > > > > 2. Pull the list of documents from my data and then get the documents > > > > from the existing ES server and put them on the new one. > > > > 3. Once all the data is in the new ES server, shutdown the older one > > > > wipe its data and then start it up as a new node to join the new ES > > > > server. > > > > > > Alternatively, instead of messing with a new server, you could: > > > > > > - create a new index 'new\_index\_timestamp' on the same ES server > > > - index from 'my\_index' to 'new\_index\_timestamp' > > > - delete 'my\_index' > > > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > > > > > The benefit of switching to using aliases is that it will make it easier > > > to make changes to your index in the future. > > > > > > clint ---
**Author:** ![Lee\_Parker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lee_parker/32/3250_2.png) [@Lee\_Parker](https://discuss.elastic.co/u/Lee_Parker)\ **Post date:** [January 20, 2011, 4:36pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/6 "2011-01-20T16:36:38Z")
We do regularly sort the results using a field we call date which contains a unix timestamp as an integer. we are already experiencing slow results from a filtered and sorted query. If we currently have about 60G of data, which we do filtered and sorted queries against regularly, should we plan to use a greater number of shards? ## Lee "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." by Marty Beckerman On Thu, Jan 20, 2011 at 10:18 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > It really depends what you plan to do with it. They can certainly be > bigger than 10gb, if you use things like facets or sorting, the values of > the field per doc will be loaded to memory, which means that you might need > more memory as the shard doc count grows, but that is linear growth, that > can be easily derived from some capacity planning down on a smaller size. > > -shay.banon > > On Thursday, January 20, 2011 at 4:01 PM, Lee Parker wrote: > > It was my understanding based on some other threads that performance will > degrade once shards are greater than 10g. If that is not true then I will > just stick with the 5 shards and 1 replica setup. > > ## Lee > > "It doesn't matter whether you are liberal or conservative, but it's > dangerous to always think with exclamation points instead of question > marks." > by Marty Beckerman > On Wed, Jan 19, 2011 at 1:22 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > The 5 shards default allow you to grow up to 5 nodes in terms of index > capacity, and in terms of improving search capacity, you can simply > dynamically add more replicas to the cluster. So, with a 5 shard 1 replica > option, you will max out on 10 nodes (nothing will be left to be allocated > to the 11th node if started). > > Sounds like you are planning on growing from 1-2 nodes to 3, this is still > well below the capacity provided by 5 shards and 1 replica. A 12gb index is > not a big one. > > -shay.banon > > On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > > Hi Lee > > In that case, here is my plan for increasing my shard count: > > 1. Spin up a new ES server but make sure it doesn't join the existing > server as a node. > 2. Pull the list of documents from my data and then get the documents > from the existing ES server and put them on the new one. > 3. Once all the data is in the new ES server, shutdown the older one > wipe its data and then start it up as a new node to join the new ES > server. > > Alternatively, instead of messing with a new server, you could: > > - create a new index 'new\_index\_timestamp' on the same ES server > - index from 'my\_index' to 'new\_index\_timestamp' > - delete 'my\_index' > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > The benefit of switching to using aliases is that it will make it easier > to make changes to your index in the future. > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 20, 2011, 4:38pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/7 "2011-01-20T16:38:37Z")
As long as you plan to add more capacity when it comes to number of servers/nodes. On Thursday, January 20, 2011 at 6:36 PM, Lee Parker wrote: > We do regularly sort the results using a field we call date which contains a unix timestamp as an integer. we are already experiencing slow results from a filtered and sorted query. If we currently have about 60G of data, which we do filtered and sorted queries against regularly, should we plan to use a greater number of shards? > > ## Lee > > "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." > by Marty Beckerman > > On Thu, Jan 20, 2011 at 10:18 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > It really depends what you plan to do with it. They can certainly be bigger than 10gb, if you use things like facets or sorting, the values of the field per doc will be loaded to memory, which means that you might need more memory as the shard doc count grows, but that is linear growth, that can be easily derived from some capacity planning down on a smaller size. > > > > -shay.banon > > On Thursday, January 20, 2011 at 4:01 PM, Lee Parker wrote: > > > > > It was my understanding based on some other threads that performance will degrade once shards are greater than 10g. If that is not true then I will just stick with the 5 shards and 1 replica setup. > > > > > > ## Lee > > > > > > "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." > > > by Marty Beckerman > > > > > > On Wed, Jan 19, 2011 at 1:22 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > The 5 shards default allow you to grow up to 5 nodes in terms of index capacity, and in terms of improving search capacity, you can simply dynamically add more replicas to the cluster. So, with a 5 shard 1 replica option, you will max out on 10 nodes (nothing will be left to be allocated to the 11th node if started). > > > > > > > > Sounds like you are planning on growing from 1-2 nodes to 3, this is still well below the capacity provided by 5 shards and 1 replica. A 12gb index is not a big one. > > > > > > > > -shay.banon > > > > On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > > > > > > > > > Hi Lee > > > > > > > > > > > In that case, here is my plan for increasing my shard count: > > > > > > > > > > > > 1. Spin up a new ES server but make sure it doesn't join the existing > > > > > > server as a node. > > > > > > 2. Pull the list of documents from my data and then get the documents > > > > > > from the existing ES server and put them on the new one. > > > > > > 3. Once all the data is in the new ES server, shutdown the older one > > > > > > wipe its data and then start it up as a new node to join the new ES > > > > > > server. > > > > > > > > > > Alternatively, instead of messing with a new server, you could: > > > > > > > > > > - create a new index 'new\_index\_timestamp' on the same ES server > > > > > - index from 'my\_index' to 'new\_index\_timestamp' > > > > > - delete 'my\_index' > > > > > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > > > > > > > > > The benefit of switching to using aliases is that it will make it easier > > > > > to make changes to your index in the future. > > > > > > > > > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 20, 2011, 4:39pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/8 "2011-01-20T16:39:43Z")
And make sure you have enough memory allocated to the nodes as well. You can check how much memory is being occupied by the field data cache by using the node stats api. On Thursday, January 20, 2011 at 6:38 PM, Shay Banon wrote: > As long as you plan to add more capacity when it comes to number of servers/nodes. > On Thursday, January 20, 2011 at 6:36 PM, Lee Parker wrote: > > > We do regularly sort the results using a field we call date which contains a unix timestamp as an integer. we are already experiencing slow results from a filtered and sorted query. If we currently have about 60G of data, which we do filtered and sorted queries against regularly, should we plan to use a greater number of shards? > > > > ## Lee > > > > "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." > > by Marty Beckerman > > > > On Thu, Jan 20, 2011 at 10:18 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > It really depends what you plan to do with it. They can certainly be bigger than 10gb, if you use things like facets or sorting, the values of the field per doc will be loaded to memory, which means that you might need more memory as the shard doc count grows, but that is linear growth, that can be easily derived from some capacity planning down on a smaller size. > > > > > > -shay.banon > > > On Thursday, January 20, 2011 at 4:01 PM, Lee Parker wrote: > > > > > > > It was my understanding based on some other threads that performance will degrade once shards are greater than 10g. If that is not true then I will just stick with the 5 shards and 1 replica setup. > > > > > > > > ## Lee > > > > > > > > "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." > > > > by Marty Beckerman > > > > > > > > On Wed, Jan 19, 2011 at 1:22 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > The 5 shards default allow you to grow up to 5 nodes in terms of index capacity, and in terms of improving search capacity, you can simply dynamically add more replicas to the cluster. So, with a 5 shard 1 replica option, you will max out on 10 nodes (nothing will be left to be allocated to the 11th node if started). > > > > > > > > > > Sounds like you are planning on growing from 1-2 nodes to 3, this is still well below the capacity provided by 5 shards and 1 replica. A 12gb index is not a big one. > > > > > > > > > > -shay.banon > > > > > On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > > > > > > > > > > > Hi Lee > > > > > > > > > > > > > In that case, here is my plan for increasing my shard count: > > > > > > > > > > > > > > 1. Spin up a new ES server but make sure it doesn't join the existing > > > > > > > server as a node. > > > > > > > 2. Pull the list of documents from my data and then get the documents > > > > > > > from the existing ES server and put them on the new one. > > > > > > > 3. Once all the data is in the new ES server, shutdown the older one > > > > > > > wipe its data and then start it up as a new node to join the new ES > > > > > > > server. > > > > > > > > > > > > Alternatively, instead of messing with a new server, you could: > > > > > > > > > > > > - create a new index 'new\_index\_timestamp' on the same ES server > > > > > > - index from 'my\_index' to 'new\_index\_timestamp' > > > > > > - delete 'my\_index' > > > > > > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > > > > > > > > > > > The benefit of switching to using aliases is that it will make it easier > > > > > > to make changes to your index in the future. > > > > > > > > > > > > clint ---
**Author:** ![Lee\_Parker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lee_parker/32/3250_2.png) [@Lee\_Parker](https://discuss.elastic.co/u/Lee_Parker)\ **Post date:** [January 20, 2011, 5:26pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/9 "2011-01-20T17:26:21Z")
The current node has 1.2g available to the heap and shows that it is using 1g of that. I will be building out to at least three nodes in the next few weeks. Is 1.2g of heap enough when we have shards of this size or larger? ## Lee "It doesn't matter whether you are liberal or conservative, but it's dangerous to always think with exclamation points instead of question marks." by Marty Beckerman On Thu, Jan 20, 2011 at 10:39 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > And make sure you have enough memory allocated to the nodes as well. You > can check how much memory is being occupied by the field data cache by using > the node stats api. > > On Thursday, January 20, 2011 at 6:38 PM, Shay Banon wrote: > > As long as you plan to add more capacity when it comes to number of > servers/nodes. > > On Thursday, January 20, 2011 at 6:36 PM, Lee Parker wrote: > > We do regularly sort the results using a field we call date which contains > a unix timestamp as an integer. we are already experiencing slow results > from a filtered and sorted query. If we currently have about 60G of data, > which we do filtered and sorted queries against regularly, should we plan to > use a greater number of shards? > > ## Lee > > "It doesn't matter whether you are liberal or conservative, but it's > dangerous to always think with exclamation points instead of question > marks." > by Marty Beckerman > On Thu, Jan 20, 2011 at 10:18 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > wrote: > > It really depends what you plan to do with it. They can certainly be > bigger than 10gb, if you use things like facets or sorting, the values of > the field per doc will be loaded to memory, which means that you might need > more memory as the shard doc count grows, but that is linear growth, that > can be easily derived from some capacity planning down on a smaller size. > > -shay.banon > > On Thursday, January 20, 2011 at 4:01 PM, Lee Parker wrote: > > It was my understanding based on some other threads that performance will > degrade once shards are greater than 10g. If that is not true then I will > just stick with the 5 shards and 1 replica setup. > > ## Lee > > "It doesn't matter whether you are liberal or conservative, but it's > dangerous to always think with exclamation points instead of question > marks." > by Marty Beckerman > On Wed, Jan 19, 2011 at 1:22 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > The 5 shards default allow you to grow up to 5 nodes in terms of index > capacity, and in terms of improving search capacity, you can simply > dynamically add more replicas to the cluster. So, with a 5 shard 1 replica > option, you will max out on 10 nodes (nothing will be left to be allocated > to the 11th node if started). > > Sounds like you are planning on growing from 1-2 nodes to 3, this is still > well below the capacity provided by 5 shards and 1 replica. A 12gb index is > not a big one. > > -shay.banon > > On Wednesday, January 19, 2011 at 7:39 PM, Clinton Gormley wrote: > > Hi Lee > > In that case, here is my plan for increasing my shard count: > > 1. Spin up a new ES server but make sure it doesn't join the existing > server as a node. > 2. Pull the list of documents from my data and then get the documents > from the existing ES server and put them on the new one. > 3. Once all the data is in the new ES server, shutdown the older one > wipe its data and then start it up as a new node to join the new ES > server. > > Alternatively, instead of messing with a new server, you could: > > - create a new index 'new\_index\_timestamp' on the same ES server > - index from 'my\_index' to 'new\_index\_timestamp' > - delete 'my\_index' > - create alias 'my\_index' pointing to 'new\_index\_timestamp' > > The benefit of switching to using aliases is that it will make it easier > to make changes to your index in the future. > > clint ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [January 20, 2011, 5:40pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/10 "2011-01-20T17:40:56Z")
is creating a new index an option? querying several indices at once is easy ... On 19 Jan., 18:29, Lee Parker [l...@klparker.com](mailto:l...@klparker.com) wrote: > We have been using Elasticsearch within our production environment for a > few months now. I have discovered that I didn't properly plan for the > amount of data in the index and find myself in need of increasing the number > of shards to improve performance. I am also going to increase the number of > nodes. I know that you can't just increase the number of shards without > reindexing the data. The difficulty in reindexing the data for me is that > our data currently resides in a highly normalized system and de-normalizing > each item into the document that ES indexes can take a long time (2.5 weeks > last time). > > In that case, here is my plan for increasing my shard count: > > 1. Spin up a new ES server but make sure it doesn't join the existing server > as a node. > 2. Pull the list of documents from my data and then get the documents from > the existing ES server and put them on the new one. > 3. Once all the data is in the new ES server, shutdown the older one wipe > its data and then start it up as a new node to join the new ES server. > > Is there a better way to keep things going and increase the shard count? We > are currently using the 5 shard default and all of the shards are around > 12G. I am thinking about setting it to 15 shards and 1 (possibly 2) > replicas. Does this make sense? We are going to have a 3 node cluster by > the end of the month and will likely increase that as we see fit throughout > the year. > > ## Lee > > "It doesn't matter whether you are liberal or conservative, but it's > dangerous to always think with exclamation points instead of question > marks." > by Marty Beckerman ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [January 20, 2011, 5:42pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/11 "2011-01-20T17:42:06Z")
ups, didn't saw shay's answers. so Lee: forget about my 😉 On 20 Jan., 18:40, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote: > is creating a new index an option? > > querying several indices at once is easy ... > > On 19 Jan., 18:29, Lee Parker [l...@klparker.com](mailto:l...@klparker.com) wrote: > > > We have been using Elasticsearch within our production environment for a > > few months now. I have discovered that I didn't properly plan for the > > amount of data in the index and find myself in need of increasing the number > > of shards to improve performance. I am also going to increase the number of > > nodes. I know that you can't just increase the number of shards without > > reindexing the data. The difficulty in reindexing the data for me is that > > our data currently resides in a highly normalized system and de-normalizing > > each item into the document that ES indexes can take a long time (2.5 weeks > > last time). > > > In that case, here is my plan for increasing my shard count: > > > > 1. Spin up a new ES server but make sure it doesn't join the existing server > > as a node. > > 2. Pull the list of documents from my data and then get the documents from > > the existing ES server and put them on the new one. > > 3. Once all the data is in the new ES server, shutdown the older one wipe > > its data and then start it up as a new node to join the new ES server. > > > Is there a better way to keep things going and increase the shard count? We > > are currently using the 5 shard default and all of the shards are around > > 12G. I am thinking about setting it to 15 shards and 1 (possibly 2) > > replicas. Does this make sense? We are going to have a 3 node cluster by > > the end of the month and will likely increase that as we see fit throughout > > the year. > > > ## Lee > > > "It doesn't matter whether you are liberal or conservative, but it's > > dangerous to always think with exclamation points instead of question > > marks." > > by Marty Beckerman ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [January 20, 2011, 5:59pm UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/12 "2011-01-20T17:59:49Z")
On Thu, 2011-01-20 at 11:26 -0600, Lee Parker wrote: > The current node has 1.2g available to the heap and shows that it is > using 1g of that. I will be building out to at least three nodes in > the next few weeks. Is 1.2g of heap enough when we have shards of > this size or larger? I would say no. In fact, I'm really surprised you haven't had out of memory issues with that amount of data and so little memory. We have 14GB of data, divided into 5 shards, one replica, and we have two nodes, each with 14GB of heap assigned to elasticsearch. The more data that can live in memory, the faster ES will perform. clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:13am UTC](https://discuss.elastic.co/t/increasing-shards-and-then-nodes/3810/13 "2017-07-06T04:13:24Z")