# Indexing performance **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2013, 5:03pm UTC](https://discuss.elastic.co/t/indexing-performance/13611 "2013-09-15T17:03:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Mingfeng\_Yang](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@Mingfeng\_Yang](https://discuss.elastic.co/u/Mingfeng_Yang)\ **Post date:** [September 15, 2013, 5:03pm UTC](https://discuss.elastic.co/t/indexing-performance/13611/1 "2013-09-15T17:03:12Z")
We are building an ES index to archive billions of docs. The index are distributed across 4 servers, each with 4 cpus, each with 32G memory. And we splitted the index into 32 shards. When we just began to index two weeks ago, the performance was like 6K docs each second. Now, as the index gets big, with 3Billion docs indexed already, total disk space is about 1.2T. The indexing speed decreases to 300 docs/second. Initially we set refresh\_interval to 120s, but in the past two days, the ES server randomly dropped one shard and the cluster health status became red. We had to decrease the refresh\_interval lower to make the cluster stable. When the cluster dropped one shard, the log has no any error info. Two questions. 1. Is there anyway to increase the performance ? 2. What's the cause of the random drop of shard? We are using ES 0.90.3. Thanks, Mingfeng -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 15, 2013, 8:10pm UTC](https://discuss.elastic.co/t/indexing-performance/13611/2 "2013-09-15T20:10:17Z")
Do you use monitoring tools, for heap/CPU/GC activity? What is the max heap size? Is heap exhausted? How large are the segments - is merging the issue? Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mingfeng\_Yang](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@Mingfeng\_Yang](https://discuss.elastic.co/u/Mingfeng_Yang)\ **Post date:** [September 15, 2013, 9:55pm UTC](https://discuss.elastic.co/t/indexing-performance/13611/3 "2013-09-15T21:55:09Z")
yes. we use bigdesk to monitor the cluster. Heap size allocated is 16G each node, and it was not exhausted. Largest .fdt file for each shard is like 2G. I just found that when a shard, say shard 17, got dropped, if I use manual allocation to put it back with this command curl -XPOST 'localhost:9200/\_cluster/reroute' -d '{"commands":[{"allocate": {"index": "boards", "shard":17, "node":"6EuzZatFRTK6q6F2boSOZw", "allow\_primary" : true}}]}' The shard is actually get deleted. How come? Ming On Sun, Sep 15, 2013 at 1:10 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Do you use monitoring tools, for heap/CPU/GC activity? What is the max > heap size? Is heap exhausted? How large are the segments - is merging the > issue? > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 15, 2013, 10:35pm UTC](https://discuss.elastic.co/t/indexing-performance/13611/4 "2013-09-15T22:35:20Z")
What OS is being used for running the ES cluster? And do you use mmapfs, maybe bootstrap mlockall? If you can't see much CPU / GC activity, it may be that your memory resources reached a limit, so OS is challenged to allocate more? Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mingfeng\_Yang](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@Mingfeng\_Yang](https://discuss.elastic.co/u/Mingfeng_Yang)\ **Post date:** [September 16, 2013, 12:25am UTC](https://discuss.elastic.co/t/indexing-performance/13611/5 "2013-09-16T00:25:24Z")
Jörg, It's centos 6.4. No customization of the kernel has been done. I will monitor the memory and see if it's related. Thanks for reminding. Ming On Sun, Sep 15, 2013 at 3:35 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > What OS is being used for running the ES cluster? And do you use mmapfs, > maybe bootstrap mlockall? > > If you can't see much CPU / GC activity, it may be that your memory > resources reached a limit, so OS is challenged to allocate more? > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Amit\_Soni](https://avatars.discourse-cdn.com/v4/letter/a/c0e974/32.png) [@Amit\_Soni](https://discuss.elastic.co/u/Amit_Soni)\ **Post date:** [October 8, 2013, 7:41pm UTC](https://discuss.elastic.co/t/indexing-performance/13611/6 "2013-10-08T19:41:11Z")
Hi Mingfeng - Just curious to know if you had any luck identifying the cause of the performance problem. -Amit. On Sun, Sep 15, 2013 at 5:25 PM, Mingfeng Yang [mfyang@wisewindow.com](mailto:mfyang@wisewindow.com)wrote: > Jörg, > > It's centos 6.4. No customization of the kernel has been done. I will > monitor the memory and see if it's related. Thanks for reminding. > > Ming > > On Sun, Sep 15, 2013 at 3:35 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > What OS is being used for running the ES cluster? And do you use mmapfs, > > maybe bootstrap mlockall? > > > > If you can't see much CPU / GC activity, it may be that your memory > > resources reached a limit, so OS is challenged to allocate more? > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:13am UTC](https://discuss.elastic.co/t/indexing-performance/13611/7 "2017-07-06T02:13:16Z")