# How many workers to parallel bulk indexing in cluster?

**URL:** <https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139>\
**Category:** Elasticsearch\
**Created:** [September 11, 2019, 8:07pm UTC](https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139 "2019-09-11T20:07:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![misam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misam/32/54016_2.png) [@misam](https://discuss.elastic.co/u/misam)\
**Post date:** [September 11, 2019, 8:07pm UTC](https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139/1 "2019-09-11T20:07:08Z")

</div>

I have a cluster with 8 nodes. I want to index a lot of docs. For the best performance,  
how many workers (to parallel bulk indexing) must run on each node?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2019, 8:52pm UTC](https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139/2 "2019-09-11T20:52:37Z")

</div>

It depends on a number of factors, e.g. document size, number of shards indexed into, mappings, hardware and bulk size. I would recommend running a benchmark with as realistic data and settings as possible.

---

<div class="post-metadata">

**Author:** ![misam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misam/32/54016_2.png) [@misam](https://discuss.elastic.co/u/misam)\
**Post date:** [September 12, 2019, 6:46am UTC](https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139/3 "2019-09-12T06:46:55Z")

</div>

If N is the best number, I must send N parallel bulk to one node or split them between all nodes?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 12, 2019, 7:08am UTC](https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139/4 "2019-09-12T07:08:43Z")

</div>

It will be the optimum for the way you tested it. I would recommend sending bulk requests to all data nodes that do indexing though.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2019, 7:08am UTC](https://discuss.elastic.co/t/how-many-workers-to-parallel-bulk-indexing-in-cluster/199139/5 "2019-10-10T07:08:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.