# Elasticsearch bad indexing timing **URL:** \ **Category:** Elasticsearch\ **Created:** [September 14, 2014, 8:38am UTC](https://discuss.elastic.co/t/elasticsearch-bad-indexing-timing/19774 "2014-09-14T08:38:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Niv\_Penso](https://avatars.discourse-cdn.com/v4/letter/n/b487fb/32.png) [@Niv\_Penso](https://discuss.elastic.co/u/Niv_Penso)\ **Post date:** [September 14, 2014, 8:38am UTC](https://discuss.elastic.co/t/elasticsearch-bad-indexing-timing/19774/1 "2014-09-14T08:38:10Z")
Hey, I am trying to migrate (copy) 35 million documents (which is a standard amount, not too big) between couchbase to elasticsearch. My elasticsearch cluster composed from 3 A3 (4 cores, 7 GB memory) CentOS Severs on Microsoft Azure (each server equals to a large server on Amazon).. I used "timing data flow" indexing to store the docuemnts. each index represents a month and composed by 3 shards and 2 replicas. when i start the migration script i see that the insertion time is becoming very slow (about 10 documents per second) and the load average of each server in the cluster jumping over than 1.5. In addition, the JVM memory is being increased almost to 100% while the cpu shows 20% and the IOps shows 20 at max. (i used Marvel CNC to get all these data) 1. Does anyone faced these kind of indexing problems in elasticsearch? 2. I would like to know if there are any parameters that i should be aware about to extend java memory? 3. is my cluster specifications good enough to handle 100 indexing per second. 4. is the indexing time depends on how big is the index? and should it be that slow? I also open a thread in stackover flow if anyone want to keep update: > Thnx Niv -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [September 14, 2014, 10:29am UTC](https://discuss.elastic.co/t/elasticsearch-bad-indexing-timing/19774/2 "2014-09-14T10:29:18Z")
A couple of suggestions: 1. Disable replicas before large amounts of inserts (set replica count to 0), and only enable it afterwards again. 2. Use batching [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html), actual batch size would depends on many factors (doc sizes, network, instances strengths) 3. Follow ES's advice on node setup, e.g. allocate 50% of the available memory size to the Java heap of ES, don't run anything else on that machine, and disable swappiness. 4. Your index is already sharded, try spreading it out to 3 different servers instead of having them on one server ("virtual shards"). This will help fan out the indexing load. 5. If you don't specify the document IDs yourself, make sure you use the latest ES, there's a significant improvement there in the ID generation mechanism which could help speeding up things. -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Sun, Sep 14, 2014 at 11:38 AM, Niv Penso [nivp@toonimo.com](mailto:nivp@toonimo.com) wrote: > Hey, > I am trying to migrate (copy) 35 million documents (which is a standard > amount, not too big) between couchbase to elasticsearch. > > My elasticsearch cluster composed from 3 A3 (4 cores, 7 GB memory) CentOS > Severs on Microsoft Azure (each server equals to a large server on Amazon).. > > I used "timing data flow" indexing to store the docuemnts. each index > represents a month and composed by 3 shards and 2 replicas. > > when i start the migration script i see that the insertion time is > becoming very slow (about 10 documents per second) and the load average of > each server in the cluster jumping over than 1.5. > In addition, the JVM memory is being increased almost to 100% while the > cpu shows 20% and the IOps shows 20 at max. > (i used Marvel CNC to get all these data) > > 1. Does anyone faced these kind of indexing problems in elasticsearch? > 2. I would like to know if there are any parameters that i should be aware > about to extend java memory? > 3. is my cluster specifications good enough to handle 100 indexing per > second. > 4. is the indexing time depends on how big is the index? and should it be > that slow? > > I also open a thread in stackover flow if anyone want to keep update: > [azure - Elasticsearch bad indexing time - Stack Overflow](http://stackoverflow.com/questions/25831518/bad-indexing-time) > > Thnx Niv > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zur6wDgSbrpnp8cVqy3mENonDXU%2BU1ZMwjq6D9P8537Lw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zur6wDgSbrpnp8cVqy3mENonDXU%2BU1ZMwjq6D9P8537Lw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Niv\_Penso](https://avatars.discourse-cdn.com/v4/letter/n/b487fb/32.png) [@Niv\_Penso](https://discuss.elastic.co/u/Niv_Penso)\ **Post date:** [September 14, 2014, 4:19pm UTC](https://discuss.elastic.co/t/elasticsearch-bad-indexing-timing/19774/3 "2014-09-14T16:19:01Z")
Amazing answer helped me so much!!! the load-avg decreased to normal number and the documents index per second increased to 85 Thnx Niv On Sunday, September 14, 2014 1:29:42 PM UTC+3, Itamar Syn-Hershko wrote: > A couple of suggestions: > > 1. Disable replicas before large amounts of inserts (set replica count to > 0), and only enable it afterwards again. > > 2. Use batching > [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html), > actual batch size would depends on many factors (doc sizes, network, > instances strengths) > > 3. Follow ES's advice on node setup, e.g. allocate 50% of the available > memory size to the Java heap of ES, don't run anything else on that > machine, and disable swappiness. > > 4. Your index is already sharded, try spreading it out to 3 different > servers instead of having them on one server ("virtual shards"). This will > help fan out the indexing load. > > 5. If you don't specify the document IDs yourself, make sure you use the > latest ES, there's a significant improvement there in the ID generation > mechanism which could help speeding up things. > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Sun, Sep 14, 2014 at 11:38 AM, Niv Penso \<[ni...@toonimo.com](mailto:ni...@toonimo.com) > \\> wrote: > > > Hey, > > I am trying to migrate (copy) 35 million documents (which is a standard > > amount, not too big) between couchbase to elasticsearch. > > > > My elasticsearch cluster composed from 3 A3 (4 cores, 7 GB memory) CentOS > > Severs on Microsoft Azure (each server equals to a large server on Amazon).. > > > > I used "timing data flow" indexing to store the docuemnts. each index > > represents a month and composed by 3 shards and 2 replicas. > > > > when i start the migration script i see that the insertion time is > > becoming very slow (about 10 documents per second) and the load average of > > each server in the cluster jumping over than 1.5. > > In addition, the JVM memory is being increased almost to 100% while the > > cpu shows 20% and the IOps shows 20 at max. > > (i used Marvel CNC to get all these data) > > > > 1. Does anyone faced these kind of indexing problems in elasticsearch? > > 2. I would like to know if there are any parameters that i should be > > aware about to extend java memory? > > 3. is my cluster specifications good enough to handle 100 indexing per > > second. > > 4. is the indexing time depends on how big is the index? and should it be > > that slow? > > > > I also open a thread in stackover flow if anyone want to keep update: > > [azure - Elasticsearch bad indexing time - Stack Overflow](http://stackoverflow.com/questions/25831518/bad-indexing-time) > > > > Thnx Niv > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37180829-62ff-4203-bfdf-fa47eeebf338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37180829-62ff-4203-bfdf-fa47eeebf338%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [September 14, 2014, 4:20pm UTC](https://discuss.elastic.co/t/elasticsearch-bad-indexing-timing/19774/4 "2014-09-14T16:20:48Z")
Sure thing -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Sun, Sep 14, 2014 at 7:19 PM, Niv Penso [nivp@toonimo.com](mailto:nivp@toonimo.com) wrote: > Amazing answer helped me so much!!! > the load-avg decreased to normal number and the documents index per second > increased to 85 > Thnx > Niv > > On Sunday, September 14, 2014 1:29:42 PM UTC+3, Itamar Syn-Hershko wrote: > > > A couple of suggestions: > > > > 1. Disable replicas before large amounts of inserts (set replica count to > > 0), and only enable it afterwards again. > > > > 2. Use batching > > [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html), > > actual batch size would depends on many factors (doc sizes, network, > > instances strengths) > > > > 3. Follow ES's advice on node setup, e.g. allocate 50% of the available > > memory size to the Java heap of ES, don't run anything else on that > > machine, and disable swappiness. > > > > 4. Your index is already sharded, try spreading it out to 3 different > > servers instead of having them on one server ("virtual shards"). This will > > help fan out the indexing load. > > > > 5. If you don't specify the document IDs yourself, make sure you use the > > latest ES, there's a significant improvement there in the ID generation > > mechanism which could help speeding up things. > > > > -- > > > > Itamar Syn-Hershko > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > > Freelance Developer & Consultant > > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > > > On Sun, Sep 14, 2014 at 11:38 AM, Niv Penso [ni...@toonimo.com](mailto:ni...@toonimo.com) wrote: > > > > > Hey, > > > I am trying to migrate (copy) 35 million documents (which is a standard > > > amount, not too big) between couchbase to elasticsearch. > > > > > > My elasticsearch cluster composed from 3 A3 (4 cores, 7 GB memory) > > > CentOS Severs on Microsoft Azure (each server equals to a large server on > > > Amazon).. > > > > > > I used "timing data flow" indexing to store the docuemnts. each index > > > represents a month and composed by 3 shards and 2 replicas. > > > > > > when i start the migration script i see that the insertion time is > > > becoming very slow (about 10 documents per second) and the load average of > > > each server in the cluster jumping over than 1.5. > > > In addition, the JVM memory is being increased almost to 100% while the > > > cpu shows 20% and the IOps shows 20 at max. > > > (i used Marvel CNC to get all these data) > > > > > > 1. Does anyone faced these kind of indexing problems in elasticsearch? > > > 2. I would like to know if there are any parameters that i should be > > > aware about to extend java memory? > > > 3. is my cluster specifications good enough to handle 100 indexing per > > > second. > > > 4. is the indexing time depends on how big is the index? and should it > > > be that slow? > > > > > > I also open a thread in stackover flow if anyone want to keep update: > > > [azure - Elasticsearch bad indexing time - Stack Overflow](http://stackoverflow.com/questions/25831518/bad-indexing-time) > > > > > > Thnx Niv > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6aee6f5f-efcb-4e37-85ef-41af2ae89a39%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/37180829-62ff-4203-bfdf-fa47eeebf338%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37180829-62ff-4203-bfdf-fa47eeebf338%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/37180829-62ff-4203-bfdf-fa47eeebf338%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37180829-62ff-4203-bfdf-fa47eeebf338%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtcwpoSUi1xmNtZDxOmfwALV0PH3n7C9Y%2B\_4U9uHGSOsA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtcwpoSUi1xmNtZDxOmfwALV0PH3n7C9Y%2B_4U9uHGSOsA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:02am UTC](https://discuss.elastic.co/t/elasticsearch-bad-indexing-timing/19774/5 "2017-07-06T01:02:29Z")