# Index creation performance **URL:** \ **Category:** Elasticsearch\ **Created:** [September 23, 2012, 9:25pm UTC](https://discuss.elastic.co/t/index-creation-performance/9112 "2012-09-23T21:25:12Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 23, 2012, 9:25pm UTC](https://discuss.elastic.co/t/index-creation-performance/9112/1 "2012-09-23T21:25:12Z")
Hi all, I'm doing some benchmarks to measure the performance of ES. 1: After storing about 1,200,000 docs(~25GB) the performance of indexing is decreasing, so I tried to create a new index after indexing each 1.200,000 docs. Each index configured to have 100 shards. Now after creating about 10 index(1000 shards), index creation performance decreases so the indexing threads are waiting for new index. 2: When I run the application just to create the indexes, the performance of index creation is ok but when at the same time it stores the data and creates the indexes I see the performance problem. I want to know that am I doing the right approach and if so how could I solve this problem. Thanks in advance, Vahid -- ---
**Author:** ![jdd](https://avatars.discourse-cdn.com/v4/letter/j/db5fbb/32.png) [@jdd](https://discuss.elastic.co/u/jdd)\ **Post date:** [September 24, 2012, 1:05am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/2 "2012-09-24T01:05:26Z")
Vahid, 100 shards per index is too many. How about trying one shard per node in the cluster? Thanks, Jaideep On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > 1,200,000 -- Jaideep Dhok -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 24, 2012, 7:28am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/3 "2012-09-24T07:28:59Z")
Hi Jaideep, thanks for your reply. I'm running one ES instance on a single node, first I want to make sure about the approach to have max performance then I will apply the configuration to the cluster. By creating an index with only one shards soon the performance of data indexing will be decreased, In addition, ES index creation speed get so slow after creating about 2000 index(one shard). During the last tests which I've run, one index with 100 shards gave me the best performance for indexing 2.4 m records(doc size ~ 22kb and the speed was 2550 rec/s), but for indexing more data the performance is decreasing(final records count is 1 billion). Thanks, Vahid On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > Vahid, > 100 shards per index is too many. How about trying one shard per node in > the cluster? > > Thanks, > Jaideep > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid \<[vhas...@gmail.com](mailto:vhas...@gmail.com) \\>wrote: > > > 1,200,000 > > -- > Jaideep Dhok -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 24, 2012, 7:59am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/4 "2012-09-24T07:59:25Z")
Problem is that with 100 shards on a single node (100 lucene instances per node) will give you High IO requests. When your index is increasing (more and more docs), Read and Write operations will cost you more. I'm pretty sure that if you run the same test on 10 nodes (10 shards per node - with replica=0), you will get best results. What I want to say here is that it's really hard to make assumptions based on what you can see on a single node. To tune ES, I recommand to do it on the target platform. It's not a linear scale approach. David. Le 24 septembre 2012 à 09:28, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) a écrit : > Hi Jaideep, thanks for your reply. > I'm running one ES instance on a single node, first I want to make sure about > the approach to have max performance then I will apply the configuration to > the cluster. > By creating an index with only one shards soon the performance of data > indexing will be decreased, In addition, ES index creation speed get so slow > after creating about 2000 index(one shard). > During the last tests which I've run, one index with 100 shards gave me the > best performance for indexing 2.4 m records(doc size ~ 22kb and the speed was > 2550 rec/s), but for indexing more data the performance is decreasing(final > records count is 1 billion). > > Thanks, > Vahid > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > > > Vahid, > > > 100 shards per index is too many. How about trying one shard per node in > > > the cluster? > > > > Thanks, > > Jaideep > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > \> \> \> 1,200,000 > > > > > > > > > > > -- > > Jaideep Dhok > > > > > > > -- -- David Pilato [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 24, 2012, 8:44am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/5 "2012-09-24T08:44:42Z")
Thank you David, With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly the result would be much more better(more hardware resources). For me the problem is that I have no measurement of ES performance. At first we were running the tests on the cluster but in the cluster there are lots of factors which affect the performance(like networking...) and performance was not acceptable and finding the bottlenecks was difficult, so I decided to find some measurements on a single node to make sure about the ES and our indexing approach and then run the tests on the cluster(anyway I do agree with you that it's not a linear scale approach). I want to find a solution to avoid decreasing the indexing performance, so I need to know that how many docs each shards can store without performance problem and if the max capacity reached what should I do. Vahid. On Mon, Sep 24, 2012 at 9:59 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > \*\* > Problem is that with 100 shards on a single node (100 lucene instances > per node) will give you High IO requests. When your index is increasing > (more and more docs), Read and Write operations will cost you more. > > I'm pretty sure that if you run the same test on 10 nodes (10 shards per > node - with replica=0), you will get best results. > > What I want to say here is that it's really hard to make assumptions > based on what you can see on a single node. To tune ES, I recommand to do > it on the target platform. > It's not a linear scale approach. > > David. > > Le 24 septembre 2012 à 09:28, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) a écrit : > > Hi Jaideep, thanks for your reply. > I'm running one ES instance on a single node, first I want to make sure > about the approach to have max performance then I will apply the > configuration to the cluster. > By creating an index with only one shards soon the performance of data > indexing will be decreased, In addition, ES index creation speed get so > slow after creating about 2000 index(one shard). > During the last tests which I've run, one index with 100 shards gave me > the best performance for indexing 2.4 m records(doc size ~ 22kb and the > speed was 2550 rec/s), but for indexing more data the performance is > decreasing(final records count is 1 billion). > > Thanks, > Vahid > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > Vahid, > 100 shards per index is too many. How about trying one shard per node in > the cluster? > > Thanks, > Jaideep > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > 1,200,000 > > -- > Jaideep Dhok > > -- > > -- > David Pilato > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > -- -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 24, 2012, 11:32am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/6 "2012-09-24T11:32:32Z")
No reply?[?] On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > Thank you David, > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly the > result would be much more better(more hardware resources). For me the > problem is that I have no measurement of ES performance. > At first we were running the tests on the cluster but in the cluster there > are lots of factors which affect the performance(like networking...) and > performance was not acceptable and finding the bottlenecks was difficult, > so I decided to find some measurements on a single node to make sure about > the ES and our indexing approach and then run the tests on the > cluster(anyway I do agree with you that it's not a linear scale approach). > > I want to find a solution to avoid decreasing the indexing performance, so > I need to know that how many docs each shards can store without performance > problem and if the max capacity reached what should I do. > > Vahid. > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > \*\* > > Problem is that with 100 shards on a single node (100 lucene instances > > per node) will give you High IO requests. When your index is increasing > > (more and more docs), Read and Write operations will cost you more. > > > > I'm pretty sure that if you run the same test on 10 nodes (10 shards per > > node - with replica=0), you will get best results. > > > > What I want to say here is that it's really hard to make assumptions > > based on what you can see on a single node. To tune ES, I recommand to do > > it on the target platform. > > It's not a linear scale approach. > > > > David. > > > > Le 24 septembre 2012 à 09:28, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) a écrit : > > > > Hi Jaideep, thanks for your reply. > > I'm running one ES instance on a single node, first I want to make sure > > about the approach to have max performance then I will apply the > > configuration to the cluster. > > By creating an index with only one shards soon the performance of data > > indexing will be decreased, In addition, ES index creation speed get so > > slow after creating about 2000 index(one shard). > > During the last tests which I've run, one index with 100 shards gave me > > the best performance for indexing 2.4 m records(doc size ~ 22kb and the > > speed was 2550 rec/s), but for indexing more data the performance is > > decreasing(final records count is 1 billion). > > > > Thanks, > > Vahid > > > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > > > Vahid, > > 100 shards per index is too many. How about trying one shard per node in > > the cluster? > > > > Thanks, > > Jaideep > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > 1,200,000 > > > > -- > > Jaideep Dhok > > > > -- > > > > -- > > David Pilato > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > -- -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 24, 2012, 12:57pm UTC](https://discuss.elastic.co/t/index-creation-performance/9112/7 "2012-09-24T12:57:41Z")
Hey Vahid Don't expect any ETA on the mailing list... Yes that's what I was meaning. What you can do, is probably to inject in a signle node (1 shard, 0 replica) and see how much docs one of your single node can handle? Then, perform the same test with 2 shards, 0 replica on the same node. Then, add a second node and perform the same test with 2 shards, 0 replica. Then, perform the same test with 4 shards, 0 replica. I think, you will be able to find the right numbers for your hardware. BTW, give a try to the wonderful Bigdesk plugin. It will help you to find some clues about IO, Memory, ... David. Le 24 septembre 2012 à 13:32, Vahid Hasani [vhasani57@gmail.com](mailto:vhasani57@gmail.com) a écrit : > No reply? > > On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani \<[vhasani57@gmail.com](mailto:vhasani57@gmail.com) > [mailto:vhasani57@gmail.com](mailto:vhasani57@gmail.com) \> wrote: > > > > Thank you David, > > > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly > > > the result would be much more better(more hardware resources). For me the > > > problem is that I have no measurement of ES performance. > > > At first we were running the tests on the cluster but in the cluster > > > there are lots of factors which affect the performance(like networking...) > > > and performance was not acceptable and finding the bottlenecks was > > > difficult, so I decided to find some measurements on a single node to make > > > sure about the ES and our indexing approach and then run the tests on the > > > cluster(anyway I do agree with you that it's not a linear scale approach). > > > > I want to find a solution to avoid decreasing the indexing performance, > > so I need to know that how many docs each shards can store without > > performance problem and if the max capacity reached what should I do. > > > > Vahid. > > > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato \<[david@pilato.fr](mailto:david@pilato.fr) > > [mailto:david@pilato.fr](mailto:david@pilato.fr) \> wrote: > > \> \> \> Problem is that with 100 shards on a single node (100 lucene > > \> \> \> instances per node) will give you High IO requests. When your > > \> \> \> index is increasing (more and more docs), Read and Write > > \> \> \> operations will cost you more. > > > > > ``` > > > I'm pretty sure that if you run the same test on 10 nodes (10 shards > > > > > > ``` > > > > > > per node - with replica=0), you will get best results. > > > > > > ``` > > > What I want to say here is that it's really hard to make assumptions > > > > > > ``` > > > > > > based on what you can see on a single node. To tune ES, I recommand to do > > > it on the target platform. > > > It's not a linear scale approach. > > > > > > ``` > > > David. > > > > > > Le 24 septembre 2012 à 09:28, Vahid < vhasani57@gmail.com > > > > > > ``` > > > > > > [mailto:vhasani57@gmail.com](mailto:vhasani57@gmail.com) \> a écrit : > > > > > > ``` > > > > > > > Hi Jaideep, thanks for your reply. > > > > > > ``` > > > > > > > ``` > > > > I'm running one ES instance on a single node, first I want to make > > > > > > > > ``` > > > > > > > > sure about the approach to have max performance then I will apply the > > > > configuration to the cluster. > > > > By creating an index with only one shards soon the performance of > > > > data indexing will be decreased, In addition, ES index creation speed > > > > get so slow after creating about 2000 index(one shard). > > > > During the last tests which I've run, one index with 100 shards > > > > gave me the best performance for indexing 2.4 m records(doc size ~ 22kb > > > > and the speed was 2550 rec/s), but for indexing more data the > > > > performance is decreasing(final records count is 1 billion). > > > > > > > > ``` > > > > Thanks, > > > > Vahid > > > > > > > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok > > > > > > > > ``` > > > > > > > > wrote: > > > > \> \> \> \> \> Vahid, > > > > > > > > > ``` > > > > > 100 shards per index is too many. How about trying one shard > > > > > > > > > > ``` > > > > > > > > > > per node in the cluster? > > > > > > > > > > ``` > > > > > Thanks, > > > > > Jaideep > > > > > > > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid > > > > > > > > > > ``` > > > > > > > > > > wrote: > > > > > \> \> \> \> \> \> 1,200,000 > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > > > > ``` > > > > > > > > > > ``` > > > > > -- > > > > > Jaideep Dhok > > > > > > > > > > > > > > > > > > > ``` > > > > > > > > ``` > > > > -- > > > > > > > > > > > > > > > > > > > ``` > > > > > > ``` > > > -- > > > David Pilato > > > http://www.scrutmydocs.org/ > > > http://dev.david.pilato.fr/ > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > -- > > > > > > ``` > > > > > > > > > > > -- -- David Pilato [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 24, 2012, 5:05pm UTC](https://discuss.elastic.co/t/index-creation-performance/9112/8 "2012-09-24T17:05:55Z")
Thank you David, @Don't expect any ETA on the mailing list... Yes,Definitely you are right, I'm a little bit under pressure... That's the exact way I've taken and these are my findings of running some tests on a machine with these config(32g Ram,8 core and 143Mb/s hdd speed): 16g Ram for ES(Xmx=Xmn), bootstrap.mlockall: true, added jvm option: -server, -XX:+AggressiveOpts for adding 500,000 docs the results were: shards: 5, 1684 docs/s shards: 50, 2336 Doc/s shards:100, 2558 I ran the tests with mores shards but the best for me was 100, so I decided to set the shards no to 100 and try with other parameters. Whats happened with more data: for 2.400.000 records the average speed was 1600 Doc/s so I started to creating new indexes after each 500,000 records and the indexing speed increased to 2558 Doc/s. Now the problem is that the final data is much more than these numbers and ES after creating 20 index (let say 2000 shards) decreases so much which in some cases creating an index takes 10 minutes. Then I was thinking that this is the maximum speed and capacity of the ES and I must add more hardware, so I asked here to make sure. Best, Vahid On Mon, Sep 24, 2012 at 2:57 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > \*\* > Hey Vahid > > Don't expect any ETA on the mailing list... > > Yes that's what I was meaning. > What you can do, is probably to inject in a signle node (1 shard, 0 > replica) and see how much docs one of your single node can handle? > Then, perform the same test with 2 shards, 0 replica on the same node. > > Then, add a second node and perform the same test with 2 shards, 0 > replica. > Then, perform the same test with 4 shards, 0 replica. > > I think, you will be able to find the right numbers for your hardware. > > BTW, give a try to the wonderful Bigdesk plugin. It will help you to find > some clues about IO, Memory, ... > > David. > > Le 24 septembre 2012 à 13:32, Vahid Hasani [vhasani57@gmail.com](mailto:vhasani57@gmail.com) a > écrit : > > No reply? > > On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani [vhasani57@gmail.com](mailto:vhasani57@gmail.com)wrote: > > Thank you David, > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly the > result would be much more better(more hardware resources). For me the > problem is that I have no measurement of ES performance. > At first we were running the tests on the cluster but in the cluster there > are lots of factors which affect the performance(like networking...) and > performance was not acceptable and finding the bottlenecks was difficult, > so I decided to find some measurements on a single node to make sure about > the ES and our indexing approach and then run the tests on the > cluster(anyway I do agree with you that it's not a linear scale approach). > > I want to find a solution to avoid decreasing the indexing performance, so > I need to know that how many docs each shards can store without performance > problem and if the max capacity reached what should I do. > > Vahid. > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > Problem is that with 100 shards on a single node (100 lucene instances > per node) will give you High IO requests. When your index is increasing > (more and more docs), Read and Write operations will cost you more. > > I'm pretty sure that if you run the same test on 10 nodes (10 shards per > node - with replica=0), you will get best results. > > What I want to say here is that it's really hard to make assumptions > based on what you can see on a single node. To tune ES, I recommand to do > it on the target platform. > It's not a linear scale approach. > > David. > > Le 24 septembre 2012 à 09:28, Vahid \< [vhasani57@gmail.com](mailto:vhasani57@gmail.com)\> a écrit : > > Hi Jaideep, thanks for your reply. > I'm running one ES instance on a single node, first I want to make sure > about the approach to have max performance then I will apply the > configuration to the cluster. > By creating an index with only one shards soon the performance of data > indexing will be decreased, In addition, ES index creation speed get so > slow after creating about 2000 index(one shard). > During the last tests which I've run, one index with 100 shards gave me > the best performance for indexing 2.4 m records(doc size ~ 22kb and the > speed was 2550 rec/s), but for indexing more data the performance is > decreasing(final records count is 1 billion). > > Thanks, > Vahid > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > Vahid, > 100 shards per index is too many. How about trying one shard per node in > the cluster? > > Thanks, > Jaideep > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > 1,200,000 > > -- > Jaideep Dhok > > -- > > -- > David Pilato > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > -- > > -- > > -- > David Pilato > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > -- -- ---
**Author:** ![BillyEm](https://avatars.discourse-cdn.com/v4/letter/b/c4cdca/32.png) [@BillyEm](https://discuss.elastic.co/u/BillyEm)\ **Post date:** [September 25, 2012, 2:03am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/9 "2012-09-25T02:03:46Z")
Vahid I think you can benefit from some alternate advice. First it is rarely necessary to start tuning on a target production machine as the process is fairly well-defined and as documented in 'A Picture of Search' applying queuing theory to capacity planning can illuminate service times that extrapolate exceedingly well to larger machine environments. That said: I suggest you review Mike's blog posts on Merge Policy and indexing performance at: > **[Visualizing Lucene's segment merges](https://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html)** > > If you've ever wondered how Lucene picks segments to merge during indexing, it looks something like this: That video displays segment merges... the following is just a snippet from one of several related posts.--\> "In fact, from the viewpoint of the MergePolicy, this is really a game against a sneaky opponent who randomly makes sudden changes to the index, such as flushing new segments or applying new deletions. If the opponent is well behaved, it'll add equal sized, large segments, which are easy to merge well, as was the case in the above video; but that's a really easy game, like playing tic-tack-toe against a 3 year old." regards, b On Monday, September 24, 2012 1:06:01 PM UTC-4, Vahid wrote: > Thank you David, > > @Don't expect any ETA on the mailing list... > Yes,Definitely you are right, I'm a little bit under pressure... > > That's the exact way I've taken and these are my findings of running some > tests on a machine with these config(32g Ram,8 core and 143Mb/s hdd speed): > 16g Ram for ES(Xmx=Xmn), bootstrap.mlockall: true, added jvm option: > -server, -XX:+AggressiveOpts > > for adding 500,000 docs the results were: > shards: 5, 1684 docs/s > shards: 50, 2336 Doc/s > shards:100, 2558 > I ran the tests with mores shards but the best for me was 100, so I > decided to set the shards no to 100 and try with other parameters. > > Whats happened with more data: for 2.400.000 records the average speed was > 1600 Doc/s so I started to creating new indexes after each 500,000 records > and the indexing speed increased to 2558 Doc/s. > Now the problem is that the final data is much more than these numbers and > ES after creating 20 index (let say 2000 shards) decreases so much which in > some cases creating an index takes 10 minutes. > Then I was thinking that this is the maximum speed and capacity of the ES > and I must add more hardware, so I asked here to make sure. > > Best, > Vahid > > On Mon, Sep 24, 2012 at 2:57 PM, David Pilato \<[da...@pilato.fr](mailto:da...@pilato.fr)\ > > > wrote: > > > \*\* > > Hey Vahid > > > > Don't expect any ETA on the mailing list... > > > > Yes that's what I was meaning. > > What you can do, is probably to inject in a signle node (1 shard, 0 > > replica) and see how much docs one of your single node can handle? > > Then, perform the same test with 2 shards, 0 replica on the same node. > > > > Then, add a second node and perform the same test with 2 shards, 0 > > replica. > > Then, perform the same test with 4 shards, 0 replica. > > > > I think, you will be able to find the right numbers for your hardware. > > > > BTW, give a try to the wonderful Bigdesk plugin. It will help you to > > find some clues about IO, Memory, ... > > > > David. > > > > Le 24 septembre 2012 à 13:32, Vahid Hasani \<[vhas...@gmail.com](mailto:vhas...@gmail.com)\\> > > a écrit : > > > > No reply? > > > > On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani \<[vhas...@gmail.com](mailto:vhas...@gmail.com)\ > > > > > wrote: > > > > Thank you David, > > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly > > the result would be much more better(more hardware resources). For me the > > problem is that I have no measurement of ES performance. > > At first we were running the tests on the cluster but in the cluster > > there are lots of factors which affect the performance(like networking...) > > and performance was not acceptable and finding the bottlenecks was > > difficult, so I decided to find some measurements on a single node to make > > sure about the ES and our indexing approach and then run the tests on the > > cluster(anyway I do agree with you that it's not a linear scale approach). > > > > I want to find a solution to avoid decreasing the indexing performance, > > so I need to know that how many docs each shards can store without > > performance problem and if the max capacity reached what should I do. > > > > Vahid. > > > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato \<[da...@pilato.fr](mailto:da...@pilato.fr)\ > > > > > wrote: > > > > Problem is that with 100 shards on a single node (100 lucene instances > > per node) will give you High IO requests. When your index is increasing > > (more and more docs), Read and Write operations will cost you more. > > > > I'm pretty sure that if you run the same test on 10 nodes (10 shards per > > node - with replica=0), you will get best results. > > > > What I want to say here is that it's really hard to make assumptions > > based on what you can see on a single node. To tune ES, I recommand to do > > it on the target platform. > > It's not a linear scale approach. > > > > David. > > > > Le 24 septembre 2012 à 09:28, Vahid \< [vhas...@gmail.com](mailto:vhas...@gmail.com) \\> a > > écrit : > > > > Hi Jaideep, thanks for your reply. > > I'm running one ES instance on a single node, first I want to make sure > > about the approach to have max performance then I will apply the > > configuration to the cluster. > > By creating an index with only one shards soon the performance of data > > indexing will be decreased, In addition, ES index creation speed get so > > slow after creating about 2000 index(one shard). > > During the last tests which I've run, one index with 100 shards gave me > > the best performance for indexing 2.4 m records(doc size ~ 22kb and the > > speed was 2550 rec/s), but for indexing more data the performance is > > decreasing(final records count is 1 billion). > > > > Thanks, > > Vahid > > > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > > > Vahid, > > 100 shards per index is too many. How about trying one shard per node in > > the cluster? > > > > Thanks, > > Jaideep > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > 1,200,000 > > > > -- > > Jaideep Dhok > > > > -- > > > > -- > > David Pilato > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > -- > > > > -- > > > > -- > > David Pilato > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > -- -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 25, 2012, 2:06pm UTC](https://discuss.elastic.co/t/index-creation-performance/9112/10 "2012-09-25T14:06:35Z")
Thanks BillyEm for your answer and good link, A few days ago I tried to play with the merge.factor and merge policy but I couldn't get good result, however I need to do more with those. But when I create a new index everything should be same as starting point?! From my understanding merge policy and such factors affect the indexing and search speed , do they affect the index creation(NOT indexing the data) performance? Best, Vahid On Tue, Sep 25, 2012 at 4:03 AM, BillyEm [wmartinusa@gmail.com](mailto:wmartinusa@gmail.com) wrote: > Vahid I think you can benefit from some alternate advice. First it is > rarely necessary to start tuning on a target production machine as the > process is fairly well-defined and as documented in 'A Picture of Search' > applying queuing theory to capacity planning can illuminate service times > that extrapolate exceedingly well to larger machine environments. That > said: > > I suggest you review Mike's blog posts on Merge Policy and indexing > performance at: > > [Changing Bits: Visualizing Lucene's segment merges](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) > > the following is just a snippet from one of several related posts.--\> > > "In fact, from the viewpoint of the MergePolicy, this is really a game > against a sneaky opponent who randomly makes sudden changes to the index, > such as flushing new segments or applying new deletions. If the opponent is > well behaved, it'll add equal sized, large segments, which are easy to > merge well, as was the case in the above video; but that's a really easy > game, like playing tic-tack-toe against a 3 year old." > > regards, > b > > On Monday, September 24, 2012 1:06:01 PM UTC-4, Vahid wrote: > > > Thank you David, > > > > @Don't expect any ETA on the mailing list... > > Yes,Definitely you are right, I'm a little bit under pressure... > > > > That's the exact way I've taken and these are my findings of running some > > tests on a machine with these config(32g Ram,8 core and 143Mb/s hdd speed): > > 16g Ram for ES(Xmx=Xmn), bootstrap.mlockall: true, added jvm option: > > -server, -XX:+AggressiveOpts > > > > for adding 500,000 docs the results were: > > shards: 5, 1684 docs/s > > shards: 50, 2336 Doc/s > > shards:100, 2558 > > I ran the tests with mores shards but the best for me was 100, so I > > decided to set the shards no to 100 and try with other parameters. > > > > Whats happened with more data: for 2.400.000 records the average speed > > was 1600 Doc/s so I started to creating new indexes after each 500,000 > > records and the indexing speed increased to 2558 Doc/s. > > Now the problem is that the final data is much more than these numbers > > and ES after creating 20 index (let say 2000 shards) decreases so much > > which in some cases creating an index takes 10 minutes. > > Then I was thinking that this is the maximum speed and capacity of the ES > > and I must add more hardware, so I asked here to make sure. > > > > Best, > > Vahid > > > > On Mon, Sep 24, 2012 at 2:57 PM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > > > \*\* > > > Hey Vahid > > > > > > Don't expect any ETA on the mailing list... > > > > > > Yes that's what I was meaning. > > > What you can do, is probably to inject in a signle node (1 shard, 0 > > > replica) and see how much docs one of your single node can handle? > > > Then, perform the same test with 2 shards, 0 replica on the same node. > > > > > > Then, add a second node and perform the same test with 2 shards, 0 > > > replica. > > > Then, perform the same test with 4 shards, 0 replica. > > > > > > I think, you will be able to find the right numbers for your hardware. > > > > > > BTW, give a try to the wonderful Bigdesk plugin. It will help you to > > > find some clues about IO, Memory, ... > > > > > > David. > > > > > > Le 24 septembre 2012 à 13:32, Vahid Hasani [vhas...@gmail.com](mailto:vhas...@gmail.com) a > > > écrit : > > > > > > No reply? > > > > > > On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani [vhas...@gmail.com](mailto:vhas...@gmail.com)wrote: > > > > > > Thank you David, > > > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly > > > the result would be much more better(more hardware resources). For me the > > > problem is that I have no measurement of ES performance. > > > At first we were running the tests on the cluster but in the cluster > > > there are lots of factors which affect the performance(like networking...) > > > and performance was not acceptable and finding the bottlenecks was > > > difficult, so I decided to find some measurements on a single node to make > > > sure about the ES and our indexing approach and then run the tests on the > > > cluster(anyway I do agree with you that it's not a linear scale approach). > > > > > > I want to find a solution to avoid decreasing the indexing performance, > > > so I need to know that how many docs each shards can store without > > > performance problem and if the max capacity reached what should I do. > > > > > > Vahid. > > > > > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > > > > Problem is that with 100 shards on a single node (100 lucene > > > instances per node) will give you High IO requests. When your index is > > > increasing (more and more docs), Read and Write operations will cost you > > > more. > > > > > > I'm pretty sure that if you run the same test on 10 nodes (10 shards > > > per node - with replica=0), you will get best results. > > > > > > What I want to say here is that it's really hard to make assumptions > > > based on what you can see on a single node. To tune ES, I recommand to do > > > it on the target platform. > > > It's not a linear scale approach. > > > > > > David. > > > > > > Le 24 septembre 2012 à 09:28, Vahid \< [vhas...@gmail.com](mailto:vhas...@gmail.com)\> a écrit : > > > > > > Hi Jaideep, thanks for your reply. > > > I'm running one ES instance on a single node, first I want to make sure > > > about the approach to have max performance then I will apply the > > > configuration to the cluster. > > > By creating an index with only one shards soon the performance of data > > > indexing will be decreased, In addition, ES index creation speed get so > > > slow after creating about 2000 index(one shard). > > > During the last tests which I've run, one index with 100 shards gave me > > > the best performance for indexing 2.4 m records(doc size ~ 22kb and the > > > speed was 2550 rec/s), but for indexing more data the performance is > > > decreasing(final records count is 1 billion). > > > > > > Thanks, > > > Vahid > > > > > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > > > > > Vahid, > > > 100 shards per index is too many. How about trying one shard per node in > > > the cluster? > > > > > > Thanks, > > > Jaideep > > > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > > 1,200,000 > > > > > > -- > > > Jaideep Dhok > > > > > > -- > > > > > > -- > > > David Pilato > > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > -- > > > > > > -- > > > > > > -- > > > David Pilato > > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > -- > > > > -- -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 25, 2012, 8:58pm UTC](https://discuss.elastic.co/t/index-creation-performance/9112/11 "2012-09-25T20:58:49Z")
Hi Vahid, index creation for itself is very fast. What you see is the overhead with 100 shards. Just use a small number of shards (default is 5 which is ok for one node). You can speed up indexing by switching off real-time indexing. Best regards Jörg On Tuesday, September 25, 2012 4:06:50 PM UTC+2, Vahid wrote: > Thanks BillyEm for your answer and good link, > > A few days ago I tried to play with the merge.factor and merge policy but > I couldn't get good result, however I need to do more with those. > But when I create a new index everything should be same as starting > point?! > From my understanding merge policy and such factors affect the indexing > and search speed , do they affect the index creation(NOT indexing the data) > performance? > > Best, > Vahid > > On Tue, Sep 25, 2012 at 4:03 AM, BillyEm \<[wmart...@gmail.com](mailto:wmart...@gmail.com) \ > > > wrote: > > > Vahid I think you can benefit from some alternate advice. First it is > > rarely necessary to start tuning on a target production machine as the > > process is fairly well-defined and as documented in 'A Picture of Search' > > applying queuing theory to capacity planning can illuminate service times > > that extrapolate exceedingly well to larger machine environments. That > > said: > > > > I suggest you review Mike's blog posts on Merge Policy and indexing > > performance at: > > > > [Changing Bits: Visualizing Lucene's segment merges](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) > > > > the following is just a snippet from one of several related posts.--\> > > > > "In fact, from the viewpoint of the MergePolicy, this is really a game > > against a sneaky opponent who randomly makes sudden changes to the index, > > such as flushing new segments or applying new deletions. If the opponent is > > well behaved, it'll add equal sized, large segments, which are easy to > > merge well, as was the case in the above video; but that's a really easy > > game, like playing tic-tack-toe against a 3 year old." > > > > regards, > > b > > > > On Monday, September 24, 2012 1:06:01 PM UTC-4, Vahid wrote: > > > > > Thank you David, > > > > > > @Don't expect any ETA on the mailing list... > > > Yes,Definitely you are right, I'm a little bit under pressure... > > > > > > That's the exact way I've taken and these are my findings of running > > > some tests on a machine with these config(32g Ram,8 core and 143Mb/s hdd > > > speed): > > > 16g Ram for ES(Xmx=Xmn), bootstrap.mlockall: true, added jvm option: > > > -server, -XX:+AggressiveOpts > > > > > > for adding 500,000 docs the results were: > > > shards: 5, 1684 docs/s > > > shards: 50, 2336 Doc/s > > > shards:100, 2558 > > > I ran the tests with mores shards but the best for me was 100, so I > > > decided to set the shards no to 100 and try with other parameters. > > > > > > Whats happened with more data: for 2.400.000 records the average speed > > > was 1600 Doc/s so I started to creating new indexes after each 500,000 > > > records and the indexing speed increased to 2558 Doc/s. > > > Now the problem is that the final data is much more than these numbers > > > and ES after creating 20 index (let say 2000 shards) decreases so much > > > which in some cases creating an index takes 10 minutes. > > > Then I was thinking that this is the maximum speed and capacity of the > > > ES and I must add more hardware, so I asked here to make sure. > > > > > > Best, > > > Vahid > > > > > > On Mon, Sep 24, 2012 at 2:57 PM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > > > > > \*\* > > > > Hey Vahid > > > > > > > > Don't expect any ETA on the mailing list... > > > > > > > > Yes that's what I was meaning. > > > > What you can do, is probably to inject in a signle node (1 shard, 0 > > > > replica) and see how much docs one of your single node can handle? > > > > Then, perform the same test with 2 shards, 0 replica on the same node. > > > > > > > > Then, add a second node and perform the same test with 2 shards, 0 > > > > replica. > > > > Then, perform the same test with 4 shards, 0 replica. > > > > > > > > I think, you will be able to find the right numbers for your hardware. > > > > > > > > BTW, give a try to the wonderful Bigdesk plugin. It will help you to > > > > find some clues about IO, Memory, ... > > > > > > > > David. > > > > > > > > Le 24 septembre 2012 à 13:32, Vahid Hasani [vhas...@gmail.com](mailto:vhas...@gmail.com) a > > > > écrit : > > > > > > > > No reply? > > > > > > > > On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani [vhas...@gmail.com](mailto:vhas...@gmail.com)wrote: > > > > > > > > Thank you David, > > > > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly > > > > the result would be much more better(more hardware resources). For me the > > > > problem is that I have no measurement of ES performance. > > > > At first we were running the tests on the cluster but in the cluster > > > > there are lots of factors which affect the performance(like networking...) > > > > and performance was not acceptable and finding the bottlenecks was > > > > difficult, so I decided to find some measurements on a single node to make > > > > sure about the ES and our indexing approach and then run the tests on the > > > > cluster(anyway I do agree with you that it's not a linear scale approach). > > > > > > > > I want to find a solution to avoid decreasing the indexing performance, > > > > so I need to know that how many docs each shards can store without > > > > performance problem and if the max capacity reached what should I do. > > > > > > > > Vahid. > > > > > > > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr)wrote: > > > > > > > > Problem is that with 100 shards on a single node (100 lucene > > > > instances per node) will give you High IO requests. When your index is > > > > increasing (more and more docs), Read and Write operations will cost you > > > > more. > > > > > > > > I'm pretty sure that if you run the same test on 10 nodes (10 shards > > > > per node - with replica=0), you will get best results. > > > > > > > > What I want to say here is that it's really hard to make assumptions > > > > based on what you can see on a single node. To tune ES, I recommand to do > > > > it on the target platform. > > > > It's not a linear scale approach. > > > > > > > > David. > > > > > > > > Le 24 septembre 2012 à 09:28, Vahid \< [vhas...@gmail.com](mailto:vhas...@gmail.com)\> a écrit : > > > > > > > > Hi Jaideep, thanks for your reply. > > > > I'm running one ES instance on a single node, first I want to make sure > > > > about the approach to have max performance then I will apply the > > > > configuration to the cluster. > > > > By creating an index with only one shards soon the performance of data > > > > indexing will be decreased, In addition, ES index creation speed get so > > > > slow after creating about 2000 index(one shard). > > > > During the last tests which I've run, one index with 100 shards gave me > > > > the best performance for indexing 2.4 m records(doc size ~ 22kb and the > > > > speed was 2550 rec/s), but for indexing more data the performance is > > > > decreasing(final records count is 1 billion). > > > > > > > > Thanks, > > > > Vahid > > > > > > > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > > > > > > > Vahid, > > > > 100 shards per index is too many. How about trying one shard per node > > > > in the cluster? > > > > > > > > Thanks, > > > > Jaideep > > > > > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > > > > 1,200,000 > > > > > > > > -- > > > > Jaideep Dhok > > > > > > > > -- > > > > > > > > -- > > > > David Pilato > > > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- > > > > David Pilato > > > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > -- > > > > > > -- -- ---
**Author:** ![anujsharma](https://avatars.discourse-cdn.com/v4/letter/a/4af34b/32.png) [@anujsharma](https://discuss.elastic.co/u/anujsharma)\ **Post date:** [September 26, 2012, 4:39am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/12 "2012-09-26T04:39:32Z")
Hi , I am creating index using ES with 8GB RAM ,1Node with 4 Shards. I am using bulk api to index my data. So i send 200 docs in one batch , my total batch is 2000.But i got performance issues ,I got java heap space exception when 20k docs get indexed. Any one suggest me how i can solve this issue. Thanks in advance ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [September 26, 2012, 7:53am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/13 "2012-09-26T07:53:59Z")
Hi Jörg, Thank you for your reply, From the tests which I've run on a single node for indexing 2,400,000(50g) I got the best performance with 100 shards and 4 index. By switching off real time indexing, if you mean disabling refresh\_interval I've already done it(I'm running the tests with -1 as refresh\_interval) however it doesn't affect performance considerably on a single node. Best, Vahid On Tue, Sep 25, 2012 at 10:58 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Hi Vahid, > > index creation for itself is very fast. What you see is the overhead with > 100 shards. Just use a small number of shards (default is 5 which is ok for > one node). You can speed up indexing by switching off real-time indexing. > > Best regards > > Jörg > > On Tuesday, September 25, 2012 4:06:50 PM UTC+2, Vahid wrote: > > > Thanks BillyEm for your answer and good link, > > > > A few days ago I tried to play with the merge.factor and merge policy but > > I couldn't get good result, however I need to do more with those. > > But when I create a new index everything should be same as starting > > point?! > > From my understanding merge policy and such factors affect the indexing > > and search speed , do they affect the index creation(NOT indexing the data) > > performance? > > > > Best, > > Vahid > > > > On Tue, Sep 25, 2012 at 4:03 AM, BillyEm [wmart...@gmail.com](mailto:wmart...@gmail.com) wrote: > > > > > Vahid I think you can benefit from some alternate advice. First it is > > > rarely necessary to start tuning on a target production machine as the > > > process is fairly well-defined and as documented in 'A Picture of Search' > > > applying queuing theory to capacity planning can illuminate service times > > > that extrapolate exceedingly well to larger machine environments. That > > > said: > > > > > > I suggest you review Mike's blog posts on Merge Policy and indexing > > > performance at: > > > > > > [http://blog.mikemccandless](http://blog.mikemccandless). **com/2011/02/visualizing-** > > > lucenes-segment-merges.html[http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html](http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html) > > > > > > the following is just a snippet from one of several related posts.--\> > > > > > > "In fact, from the viewpoint of the MergePolicy, this is really a game > > > against a sneaky opponent who randomly makes sudden changes to the index, > > > such as flushing new segments or applying new deletions. If the opponent is > > > well behaved, it'll add equal sized, large segments, which are easy to > > > merge well, as was the case in the above video; but that's a really easy > > > game, like playing tic-tack-toe against a 3 year old." > > > > > > regards, > > > b > > > > > > On Monday, September 24, 2012 1:06:01 PM UTC-4, Vahid wrote: > > > > > > > Thank you David, > > > > > > > > @Don't expect any ETA on the mailing list... > > > > Yes,Definitely you are right, I'm a little bit under pressure... > > > > > > > > That's the exact way I've taken and these are my findings of running > > > > some tests on a machine with these config(32g Ram,8 core and 143Mb/s hdd > > > > speed): > > > > 16g Ram for ES(Xmx=Xmn), bootstrap.mlockall: true, added jvm option: > > > > -server, -XX:+AggressiveOpts > > > > > > > > for adding 500,000 docs the results were: > > > > shards: 5, 1684 docs/s > > > > shards: 50, 2336 Doc/s > > > > shards:100, 2558 > > > > I ran the tests with mores shards but the best for me was 100, so I > > > > decided to set the shards no to 100 and try with other parameters. > > > > > > > > Whats happened with more data: for 2.400.000 records the average speed > > > > was 1600 Doc/s so I started to creating new indexes after each 500,000 > > > > records and the indexing speed increased to 2558 Doc/s. > > > > Now the problem is that the final data is much more than these numbers > > > > and ES after creating 20 index (let say 2000 shards) decreases so much > > > > which in some cases creating an index takes 10 minutes. > > > > Then I was thinking that this is the maximum speed and capacity of the > > > > ES and I must add more hardware, so I asked here to make sure. > > > > > > > > Best, > > > > Vahid > > > > > > > > On Mon, Sep 24, 2012 at 2:57 PM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > > > > > > > > > \*\* > > > > > Hey Vahid > > > > > > > > > > Don't expect any ETA on the mailing list... > > > > > > > > > > Yes that's what I was meaning. > > > > > What you can do, is probably to inject in a signle node (1 shard, 0 > > > > > replica) and see how much docs one of your single node can handle? > > > > > Then, perform the same test with 2 shards, 0 replica on the same > > > > > node. > > > > > > > > > > Then, add a second node and perform the same test with 2 shards, 0 > > > > > replica. > > > > > Then, perform the same test with 4 shards, 0 replica. > > > > > > > > > > I think, you will be able to find the right numbers for your > > > > > hardware. > > > > > > > > > > BTW, give a try to the wonderful Bigdesk plugin. It will help you to > > > > > find some clues about IO, Memory, ... > > > > > > > > > > David. > > > > > > > > > > Le 24 septembre 2012 à 13:32, Vahid Hasani [vhas...@gmail.com](mailto:vhas...@gmail.com) a > > > > > écrit : > > > > > > > > > > No reply? > > > > > > > > > > On Mon, Sep 24, 2012 at 10:44 AM, Vahid Hasani [vhas...@gmail.com](mailto:vhas...@gmail.com)wrote: > > > > > > > > > > Thank you David, > > > > > With 10 nodes you mean 10 ES instance on 10 machines? if so, certainly > > > > > the result would be much more better(more hardware resources). For me the > > > > > problem is that I have no measurement of ES performance. > > > > > At first we were running the tests on the cluster but in the cluster > > > > > there are lots of factors which affect the performance(like networking...) > > > > > and performance was not acceptable and finding the bottlenecks was > > > > > difficult, so I decided to find some measurements on a single node to make > > > > > sure about the ES and our indexing approach and then run the tests on the > > > > > cluster(anyway I do agree with you that it's not a linear scale approach). > > > > > > > > > > I want to find a solution to avoid decreasing the indexing > > > > > performance, so I need to know that how many docs each shards can store > > > > > without performance problem and if the max capacity reached what should I > > > > > do. > > > > > > > > > > Vahid. > > > > > > > > > > On Mon, Sep 24, 2012 at 9:59 AM, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr)wrote: > > > > > > > > > > Problem is that with 100 shards on a single node (100 lucene > > > > > instances per node) will give you High IO requests. When your index is > > > > > increasing (more and more docs), Read and Write operations will cost you > > > > > more. > > > > > > > > > > I'm pretty sure that if you run the same test on 10 nodes (10 shards > > > > > per node - with replica=0), you will get best results. > > > > > > > > > > What I want to say here is that it's really hard to make assumptions > > > > > based on what you can see on a single node. To tune ES, I recommand to do > > > > > it on the target platform. > > > > > It's not a linear scale approach. > > > > > > > > > > David. > > > > > > > > > > Le 24 septembre 2012 à 09:28, Vahid \< [vhas...@gmail.com](mailto:vhas...@gmail.com)\> a écrit : > > > > > > > > > > Hi Jaideep, thanks for your reply. > > > > > I'm running one ES instance on a single node, first I want to make > > > > > sure about the approach to have max performance then I will apply the > > > > > configuration to the cluster. > > > > > By creating an index with only one shards soon the performance of data > > > > > indexing will be decreased, In addition, ES index creation speed get so > > > > > slow after creating about 2000 index(one shard). > > > > > During the last tests which I've run, one index with 100 shards gave > > > > > me the best performance for indexing 2.4 m records(doc size ~ 22kb and the > > > > > speed was 2550 rec/s), but for indexing more data the performance is > > > > > decreasing(final records count is 1 billion). > > > > > > > > > > Thanks, > > > > > Vahid > > > > > > > > > > On Monday, September 24, 2012 3:05:50 AM UTC+2, jaideep dhok wrote: > > > > > > > > > > Vahid, > > > > > 100 shards per index is too many. How about trying one shard per node > > > > > in the cluster? > > > > > > > > > > Thanks, > > > > > Jaideep > > > > > > > > > > On Mon, Sep 24, 2012 at 2:55 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > > > > > > 1,200,000 > > > > > > > > > > -- > > > > > Jaideep Dhok > > > > > > > > > > -- > > > > > > > > > > -- > > > > > David Pilato > > > > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > -- > > > > > David Pilato > > > > > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) > > > > > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > > > -- > > > > > > > > -- > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:11am UTC](https://discuss.elastic.co/t/index-creation-performance/9112/14 "2017-07-06T03:11:22Z")