# Bulk indexing and number of shards **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2014, 8:05pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979 "2014-04-12T20:05:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![eunever32](https://avatars.discourse-cdn.com/v4/letter/e/e68b1a/32.png) [@eunever32](https://discuss.elastic.co/u/eunever32)\ **Post date:** [April 12, 2014, 8:05pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979/1 "2014-04-12T20:05:22Z")
Hi, I'm testing on a single node. I find I can get better bulk indexing performance when the index has more shards. Does that make sense ? My own theory is that when I have multiple bulk clients, then by increasing shards the server achieves better concurrency (?) So if I increase the shards to say 30, and get a good indexing run... is it possible to reduce the number of shards subsequently.. or does it matter if the number remains at say 30? Thanks, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 13, 2014, 11:30pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979/2 "2014-04-13T23:30:09Z")
Sorry, you can't reduce it. I imagine the performance increase you get is because the merge logic is per shard so it does less when there are more shards for the same data. You can likely get similar numbers if you set the refresh interval to -1 and play with the merge policy before the bulk load. You'd want to reset it afterwords and then run an optimize. This amounts to the same thing as starting with more shards and merging them. Mostly. I think. Sent from my iPhone > On Apr 12, 2014, at 4:05 PM, [eunever32@gmail.com](mailto:eunever32@gmail.com) wrote: > > Hi, > > I'm testing on a single node. > > I find I can get better bulk indexing performance when the index has more shards. Does that make sense ? > > My own theory is that when I have multiple bulk clients, then by increasing shards the server achieves better concurrency (?) > > So if I increase the shards to say 30, and get a good indexing run... is it possible to reduce the number of shards subsequently.. or does it matter if the number remains at say 30? > > Thanks, > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 14, 2014, 7:08am UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979/3 "2014-04-14T07:08:31Z")
More thoughts in addition to Nik: - the default setting is refresh by every second. Refresh works very fast when segments are small. If you have more than one shard and use bulk indexing, the segments are small enough for refresh for a longer time. So you will observe a faster bulk indexing, but only for the first 20 minutes or so (longer runs will also show increasing response times). Disabling refresh at bulk indexing time is strongly recommended. - the default, ES settings are selected for more than one shard (default is 5). To utilize the server resources (CPU, RAM) by a single shard, a little optimization of thread pools and buffer sizes may be required, especially the bulk thread pool and the index buffer size. Jörg On Mon, Apr 14, 2014 at 1:30 AM, Nik Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > Sorry, you can't reduce it. I imagine the performance increase you get is > because the merge logic is per shard so it does less when there are more > shards for the same data. You can likely get similar numbers if you set the > refresh interval to -1 and play with the merge policy before the bulk load. > You'd want to reset it afterwords and then run an optimize. This amounts to > the same thing as starting with more shards and merging them. Mostly. I > think. > > Sent from my iPhone > > On Apr 12, 2014, at 4:05 PM, [eunever32@gmail.com](mailto:eunever32@gmail.com) wrote: > > Hi, > > I'm testing on a single node. > > I find I can get better bulk indexing performance when the index has more > shards. Does that make sense ? > > My own theory is that when I have multiple bulk clients, then by > increasing shards the server achieves better concurrency (?) > > So if I increase the shards to say 30, and get a good indexing run... is > it possible to reduce the number of shards subsequently.. or does it matter > if the number remains at say 30? > > Thanks, > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d6006698-0ced-43f4-959f-52def820013f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com)[https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0CCF99CF-B24F-4C39-9C12-703039EB5BB1%40gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEsXc5BOuC%2B85TwpSz%3DV\_9RACUHmUnA269uW-194uPjA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEsXc5BOuC%2B85TwpSz%3DV_9RACUHmUnA269uW-194uPjA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![eunever32](https://avatars.discourse-cdn.com/v4/letter/e/e68b1a/32.png) [@eunever32](https://discuss.elastic.co/u/eunever32)\ **Post date:** [April 14, 2014, 5:04pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979/4 "2014-04-14T17:04:11Z")
Thanks guys, yes I already have refresh interval at -1 What I'm suggesting is that to support multiple client threads say : 50 then it seems that 50 shards is a big help. ie more shards equals more concurrency. Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1b531518-ee7c-424d-a238-eb92c3e59d6d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1b531518-ee7c-424d-a238-eb92c3e59d6d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 14, 2014, 6:36pm UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979/5 "2014-04-14T18:36:45Z")
Although many shards mean higher concurrency, a single shard also has very high concurrency. ES concurrency is implemented independent of the number of shards and is very flexible in configuration of thread pools and modules. The default settings are for a small ES system which can scale to a around 5 nodes, so if you have a powerful machine, you will see that more than one shard gives better experience simply because the default settings are not designed for being limited to a single node. 50 shards per node is quite high and needs heavy adjustment at other places unless you go for a total number of around 50 nodes in the cluster, so I do not recommend high number of shards by default. Jörg On Mon, Apr 14, 2014 at 7:04 PM, [eunever32@gmail.com](mailto:eunever32@gmail.com) wrote: > Thanks guys, > yes I already have refresh interval at -1 > > What I'm suggesting is that to support multiple client threads say : 50 > then it seems that 50 shards is a big help. > ie more shards equals more concurrency. > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1b531518-ee7c-424d-a238-eb92c3e59d6d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1b531518-ee7c-424d-a238-eb92c3e59d6d%40googlegroups.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHoRxLGze22d8PgrjsmWff6FndAqafZx%2BN5J3WSd%3DSYVw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHoRxLGze22d8PgrjsmWff6FndAqafZx%2BN5J3WSd%3DSYVw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:35am UTC](https://discuss.elastic.co/t/bulk-indexing-and-number-of-shards/16979/6 "2017-07-06T01:35:53Z")