# Intial load and Cluster configuration query **URL:** \ **Category:** Elasticsearch\ **Created:** [May 19, 2011, 5:32pm UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438 "2011-05-19T17:32:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![lalit\_mishra](https://avatars.discourse-cdn.com/v4/letter/l/b77776/32.png) [@lalit\_mishra](https://discuss.elastic.co/u/lalit_mishra)\ **Post date:** [May 19, 2011, 5:32pm UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438/1 "2011-05-19T17:32:50Z")
Hi All, I have below queries pertaining to elastic search 1. In our application we would like to index around 2-3 million documents what should be the best cluster/Shard configuration ? 2. Our initial load consists of indexing 2-3 million records what are configuration we should make in elastic search so that we achieve faster indexing for initial load ? 3. Currenlty we have a batch process which spawns 10 threads this threads sends request over to elastic search server? Do we forsee any issue with this approach (like write lock)? Thanks for your help. Regards, Lalit. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 19, 2011, 8:14pm UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438/2 "2011-05-19T20:14:44Z")
On Thursday, May 19, 2011 at 8:32 PM, lalit mishra wrote: Hi All, > I have below queries pertaining to Elasticsearch > > 1. In our application we would like to index around 2-3 million documents what should be the best cluster/Shard configuration ? Hard to tell, since I don't know the size of the documents. You will need to do some capacity planning. Generally, the default 5 shards should be more than enough for 2-3 million, and depending on the docs, you can even use lower number of shards (will use less memory). > 1. Our initial load consists of indexing 2-3 million records what are configuration we should make in Elasticsearch so that we achieve faster indexing for initial load ? Indexing rate really depends on many factors. As for scaling out, the default 5 shards set for an index will mean you can grow upto 5 machines. If oyu have 1 replica (the default), then you can grow up to 10 machines without hitting a wall. > 1. Currenlty we have a batch process which spawns 10 threads this threads sends request over to Elasticsearch server? Do we forsee any issue with this approach (like write lock)? No, no issues here. Check that your indexing machine is not the bottleneck. > Thanks for your help. > > Regards, > Lalit. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [May 20, 2011, 7:46pm UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438/3 "2011-05-20T19:46:59Z")
> 1. Our initial load consists of indexing 2-3 million records what are > configuration we should make in Elasticsearch so that we achieve faster > indexing for initial load ? hopefully I'm not wrong with this, but you can increase the real time latency (refresh\_interval) which improves indexing. @Shay: Is that a correct assumption? Also tuning lucene's merge factor (increase it) can improve indexing speed (but you pay querying time). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 20, 2011, 7:52pm UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438/4 "2011-05-20T19:52:53Z")
You can change those in real time, check the update settings API which has a sample for it: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html). On Friday, May 20, 2011 at 10:46 PM, Karussell wrote: > 1. Our initial load consists of indexing 2-3 million records what are > > > configuration we should make in Elasticsearch so that we achieve faster > > indexing for initial load ? > > hopefully I'm not wrong with this, but you can increase the real time > latency (refresh\_interval) which improves indexing. @Shay: Is that a > correct assumption? > > Also tuning lucene's merge factor (increase it) can improve indexing > speed (but you pay querying time). ---
**Author:** ![lalit\_mishra](https://avatars.discourse-cdn.com/v4/letter/l/b77776/32.png) [@lalit\_mishra](https://discuss.elastic.co/u/lalit_mishra)\ **Post date:** [May 21, 2011, 1:41am UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438/5 "2011-05-21T01:41:09Z")
Hi Shay, So I need to use below configuration during initial load of the records curl -XPUT localhost:9200/test/\_settings -d '{ "index" : { "refresh\_interval" : "-1", "merge.policy.merge\_factor" : 30 } }' Once the initial load is over then revert the indices settings to curl -XPUT localhost:9200/test/\_settings -d '{ "index" : { "refresh\_interval" : "1s", "merge.policy.merge\_factor" : 10 } }' curl -XPOST '[http://localhost:9200/test/\_optimize?max\_num\_segments=5](http://localhost:9200/test/_optimize?max_num_segments=5)' Hope this should further speed up the process. Thanks, Lalit. On Sat, May 21, 2011 at 1:22 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > You can change those in real time, check the update settings API which > has a sample for it: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html) > . > > On Friday, May 20, 2011 at 10:46 PM, Karussell wrote: > > 1. Our initial load consists of indexing 2-3 million records what are > configuration we should make in Elasticsearch so that we achieve faster > indexing for initial load ? > > hopefully I'm not wrong with this, but you can increase the real time > latency (refresh\_interval) which improves indexing. @Shay: Is that a > correct assumption? > > Also tuning lucene's merge factor (increase it) can improve indexing > speed (but you pay querying time). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:05am UTC](https://discuss.elastic.co/t/intial-load-and-cluster-configuration-query/4438/6 "2017-07-06T04:05:40Z")