# Few question about elasticsearch index and cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2012, 10:51am UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444 "2012-10-23T10:51:15Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Robbie\_Cheng](https://avatars.discourse-cdn.com/v4/letter/r/d26b3c/32.png) [@Robbie\_Cheng](https://discuss.elastic.co/u/Robbie_Cheng)\ **Post date:** [October 23, 2012, 10:51am UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444/1 "2012-10-23T10:51:15Z")
-can we add/remove specific fields dynamically, once the index has been created? -can we configure ratio of index in memory or in file? -can we add new node into elasticsearch cluster when it is required? any side effect? -- ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [October 23, 2012, 11:57am UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444/2 "2012-10-23T11:57:58Z")
- You can add new fields easily, but they cannot be removed without reindexing - Not sure what you mean by ratio. Elasticsearch provides some control over individual cache sizes[http://www.elasticsearch.org/guide/reference/index-modules/cache.html](http://www.elasticsearch.org/guide/reference/index-modules/cache.html). It also allows to allocate indices completely in memory or memory map your indices using mmapfs index store.[http://www.elasticsearch.org/guide/reference/index-modules/store.html](http://www.elasticsearch.org/guide/reference/index-modules/store.html) - You can add and remove nodes from elasticsearch cluster. When a new node is added to the cluster, elasticsearch rebalances its shards to make sure that all nodes in the cluster have the same number of shards. On Tuesday, October 23, 2012 6:51:19 AM UTC-4, Robbie Cheng wrote: > -can we add/remove specific fields dynamically, once the index has > been created? > -can we configure ratio of index in memory or in file? > -can we add new node into elasticsearch cluster when it is required? > any side effect? -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [October 23, 2012, 12:12pm UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444/3 "2012-10-23T12:12:35Z")
Hello Robbie, On Tue, Oct 23, 2012 at 1:51 PM, Robbie Cheng [robbiecheng@gmail.com](mailto:robbiecheng@gmail.com) wrote: > -can we add/remove specific fields dynamically, once the index has > been created? You can always add new fields to your mapping, but you can't remove them. If you want to remove a field you will have to reindex your data. For more information about mapping, take a look here: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... > -can we configure ratio of index in memory or in file? I'm not sure if I got the question right, but I don't think you can. You can specify whether to store your index in memory or on the file system, and there are some settings for each: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... Given your question, I suppose your data doesn't fit in memory, so you'd probably want to store your indices on the file system, and leave it up to the OS to cache some of that in memory. Elasticsearch also has its own caches: for example, many filters are cached by default and you can change settings there. You can find some more information about ES caching here: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... > -can we add new node into elasticsearch cluster when it is required? > any side effect? Definitely. This is where sharding comes into play. In short, it goes like this: - each index is divided into shards. By default there are 5, but you can change that for each index - each shard might have a number of replicas. Replicas are good for redundancy and also query performance (since queries will also run on replicas). By default there's 1 replica for each shard So if you have one node and one index, by default you can see all your 5 shards allocated on your unique node. Replicas won't get allocated in this scenario because it doesn't make sense. If you add one more node, the 5 replicas will get assigned to it. If you continue to add nodes, your 5 shards and 5 replicas (one per shard) will be automatically balanced between nodes so that each node will get roughly the same number of indices. For example, on 5 nodes you'll get 2 shards for each (whether they are primary shards or replicas). So with one index, with the default configuration, you're good for 10 nodes. If you add an 11th node, it will get no shards. While you can change the number of replicas on a live cluster, you can't change the number of primary shards. So that's why you'd need to plan your number of shards in production. Please note that the logic above applies to the total number of shards in your cluster. For all your indices. So, for example if you have 2 indices with the default configuration on a cluster, Elasticsearch can distribute your data on up to 20 nodes. ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![Robbie\_Cheng](https://avatars.discourse-cdn.com/v4/letter/r/d26b3c/32.png) [@Robbie\_Cheng](https://discuss.elastic.co/u/Robbie_Cheng)\ **Post date:** [November 22, 2012, 1:17am UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444/4 "2012-11-22T01:17:47Z")
Hey Radu, Thanks for your detailed reply, one follow-up questions about the # of primary shards since it can't be change afterwards? What's the maxium number of primary shards, and is there any way that we can determine how many of primary shards we need? Thanks, Radu Gheorghe於 2012年10月23日星期二UTC+8下午8時12分39秒寫道： > Hello Robbie, > > On Tue, Oct 23, 2012 at 1:51 PM, Robbie Cheng \<[robbi...@gmail.com](mailto:robbi...@gmail.com)\\> > wrote: > > > -can we add/remove specific fields dynamically, once the index has > > been created? > > You can always add new fields to your mapping, but you can't remove > them. If you want to remove a field you will have to reindex your > data. > > For more information about mapping, take a look here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/) > > > -can we configure ratio of index in memory or in file? > > I'm not sure if I got the question right, but I don't think you can. > You can specify whether to store your index in memory or on the file > system, and there are some settings for each: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > Given your question, I suppose your data doesn't fit in memory, so > you'd probably want to store your indices on the file system, and > leave it up to the OS to cache some of that in memory. Elasticsearch > also has its own caches: for example, many filters are cached by > default and you can change settings there. You can find some more > information about ES caching here: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/cache.html) > > > -can we add new node into elasticsearch cluster when it is required? > > any side effect? > > Definitely. This is where sharding comes into play. In short, it goes like > this: > > - each index is divided into shards. By default there are 5, but you > can change that for each index > - each shard might have a number of replicas. Replicas are good for > redundancy and also query performance (since queries will also run on > replicas). By default there's 1 replica for each shard > > So if you have one node and one index, by default you can see all your > 5 shards allocated on your unique node. Replicas won't get allocated > in this scenario because it doesn't make sense. > > If you add one more node, the 5 replicas will get assigned to it. > > If you continue to add nodes, your 5 shards and 5 replicas (one per > shard) will be automatically balanced between nodes so that each node > will get roughly the same number of indices. For example, on 5 nodes > you'll get 2 shards for each (whether they are primary shards or > replicas). > > So with one index, with the default configuration, you're good for 10 > nodes. If you add an 11th node, it will get no shards. While you can > change the number of replicas on a live cluster, you can't change the > number of primary shards. So that's why you'd need to plan your number > of shards in production. > > Please note that the logic above applies to the total number of shards > in your cluster. For all your indices. So, for example if you have 2 > indices with the default configuration on a cluster, Elasticsearch can > distribute your data on up to 20 nodes. > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [November 24, 2012, 10:16am UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444/5 "2012-11-24T10:16:58Z")
Hello Robbie, On Thu, Nov 22, 2012 at 3:17 AM, Robbie Cheng [robbiecheng@gmail.com](mailto:robbiecheng@gmail.com) wrote: > Hey Radu, > > Thanks for your detailed reply, one follow-up questions about the # of > primary shards since it can't be change afterwards? > What's the maxium number of primary shards, and is there any way that we > can determine how many of primary shards we need? If you're going to have a static number of indices, you need to think on how many nodes you're going to split a single set of those indices (without accounting replicas) in the log run, without having to reindex your data. But you also have to account that each shard comes with an overhead, so you can't just go with 1000 shards, "just to be sure". Here's a very good video in which you can see some solutions on how you can organize your indices and shards: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:03am UTC](https://discuss.elastic.co/t/few-question-about-elasticsearch-index-and-cluster/9444/6 "2017-07-06T03:03:03Z")