# Indexing performance over index size **URL:** \ **Category:** Elasticsearch\ **Created:** [July 30, 2013, 9:13am UTC](https://discuss.elastic.co/t/indexing-performance-over-index-size/12999 "2013-07-30T09:13:00Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Harris](https://avatars.discourse-cdn.com/v4/letter/h/c37758/32.png) [@Harris](https://discuss.elastic.co/u/Harris)\ **Post date:** [July 30, 2013, 9:13am UTC](https://discuss.elastic.co/t/indexing-performance-over-index-size/12999/1 "2013-07-30T09:13:00Z")
After performing quite a lot of experiments on a 5 node ES cluster, I noticed that performance is heavily impacted by the index size. [https://lh6.googleusercontent.com/-G24uCWlCxPc/UfeCFJCmm5I/AAAAAAAAABM/r5Bxda-XkXo/s1600/ES\_slope.png](https://lh6.googleusercontent.com/-G24uCWlCxPc/UfeCFJCmm5I/AAAAAAAAABM/r5Bxda-XkXo/s1600/ES_slope.png) So when I performed bulk indexing into some empty indexes, this was the performance I go (check img above). If I understand well, performance is not affected by other big indexes that exist on the cluster but are not getting indexed at the time. Thus, is it best to perform indexing into a new index every day or week using aliases in order to manage them? Given the fact that at the same time, I want to keep performing search queries on all the indexes (current and past ones) does the above make any sense to do? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 30, 2013, 1:17pm UTC](https://discuss.elastic.co/t/indexing-performance-over-index-size/12999/2 "2013-07-30T13:17:08Z")
How long is the duration you performed the indexing? What VM, what settings of ES? In my experiments, I can index for hours, and the index throughput scales well with the number of nodes (where the remote feed client is mostly network and CPU bound). Maybe you measured Lucene segment merging activity? Because this is what takes additional time when index grows, compared to an empty index, not the index size. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Harris](https://avatars.discourse-cdn.com/v4/letter/h/c37758/32.png) [@Harris](https://discuss.elastic.co/u/Harris)\ **Post date:** [July 30, 2013, 3:52pm UTC](https://discuss.elastic.co/t/indexing-performance-over-index-size/12999/3 "2013-07-30T15:52:45Z")
On a 5 node cluster (4 cores, 4G Ram per node), we had 4 node-clients continuously bulk indexing data for 2-3 days. The biggest change is on the first 10 hours or so where performance declines rapidly 2 of the appls did bulk indexing on separate indexes while the 2 other did it on the same index. On the first 10 hours we had indexed 25G and totally about 45 index at the end of our experiment. On Tue, Jul 30, 2013 at 4:17 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > How long is the duration you performed the indexing? What VM, what > settings of ES? > > In my experiments, I can index for hours, and the index throughput scales > well with the number of nodes (where the remote feed client is mostly > network and CPU bound). > > Maybe you measured Lucene segment merging activity? Because this is what > takes additional time when index grows, compared to an empty index, not the > index size. > > Jörg > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/TVjxjMCJSp4/unsubscribe](https://groups.google.com/d/topic/elasticsearch/TVjxjMCJSp4/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 30, 2013, 5:16pm UTC](https://discuss.elastic.co/t/indexing-performance-over-index-size/12999/4 "2013-07-30T17:16:29Z")
Did you use custom settings for index refresh, segment merging, JVM GC? The default values [http://www.elasticsearch.org/guide/reference/index-modules/merge/](http://www.elasticsearch.org/guide/reference/index-modules/merge/) are a bit straight for a 2G heap and long-time bulk indexing. I would suggest 1G for max\_merged\_segment and 20 segments\_per\_tier for more consistent performance on your configuration, together with disabled index refreshing and GC tuning. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:23am UTC](https://discuss.elastic.co/t/indexing-performance-over-index-size/12999/5 "2017-07-06T02:23:42Z")
Closed on July 6, 2017, 2:23am UTC