# How to run multiple indexing threads on a single machine?

**URL:** <https://discuss.elastic.co/t/how-to-run-multiple-indexing-threads-on-a-single-machine/311058>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [July 31, 2022, 3:08pm UTC](https://discuss.elastic.co/t/how-to-run-multiple-indexing-threads-on-a-single-machine/311058 "2022-07-31T15:08:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Niels\_Basjes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_basjes/32/109091_2.png) [@Niels\_Basjes](https://discuss.elastic.co/u/Niels_Basjes)\
**Post date:** [July 31, 2022, 3:08pm UTC](https://discuss.elastic.co/t/how-to-run-multiple-indexing-threads-on-a-single-machine/311058/1 "2022-07-31T15:08:05Z")

</div>

I have written an Elasticsearch plugin ( analyzes the UserAgent string ) that works the way I have in mind when I put it in a pipeline. (see [Elastic Search | Yauaa - Yet Another UserAgent Analyzer](https://yauaa.basjes.nl/udf/elastic-search/) )

Now on my laptop I want to put a rather large dataset (~100M records) in Elasticsearch via this plugin and check the results in Kibana.  
I have put together some scripting ( [yauaa/devtools/analysis at main · nielsbasjes/yauaa · GitHub](https://github.com/nielsbasjes/yauaa/tree/main/devtools/analysis) ) that starts Elasticsearch with the plugin installed using Docker on my Ubuntu machine.  
I then define the pipeline and load the data.

Functionally this works.

The problem I have is that when I do this I have been unable to make this pipeline run in multiple threads and thus reduce the time I have to wait. At the moment it is only using ~2-3 cpu cores where my laptop has 12 (6+hyperthreading).  
I am doing 8 bulk updates at a time with ~ 100000 records in each batch.

What config setting can I change to get ES to actually use multiple threads for this pipeline so that it uses ~10 CPU cores?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 1, 2022, 10:33pm UTC](https://discuss.elastic.co/t/how-to-run-multiple-indexing-threads-on-a-single-machine/311058/2 "2022-08-01T22:33:27Z")

</div>

The only way I can think to influence this would be to run multiple nodes on the host.

Maybe someone else can comment though?

---

<div class="post-metadata">

**Author:** ![Niels\_Basjes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niels_basjes/32/109091_2.png) [@Niels\_Basjes](https://discuss.elastic.co/u/Niels_Basjes)\
**Post date:** [August 3, 2022, 8:44am UTC](https://discuss.elastic.co/t/how-to-run-multiple-indexing-threads-on-a-single-machine/311058/3 "2022-08-03T08:44:54Z")

</div>

This clarifies a lot.  
I'm going to try a different approach now.  
Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 31, 2022, 8:45am UTC](https://discuss.elastic.co/t/how-to-run-multiple-indexing-threads-on-a-single-machine/311058/4 "2022-08-31T08:45:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.