# Multiple child inastances of a single client or multiple clients, which is better for bulk indexing in large rates? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 18, 2023, 3:55am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293 "2023-06-18T03:55:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![shameel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shameel/32/122398_2.png) [@shameel](https://discuss.elastic.co/u/shameel)\ **Post date:** [June 18, 2023, 3:55am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/1 "2023-06-18T03:55:56Z")
Hi Im using Elasticsearch v7.5.0 and I have a huge number of documents being ingested per second, as per the documentation it is recommended to use multiple clients for bulk indexing to reduce load. Can I get the same results if I'm using multiple child instances of a single client? [child clients](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/child.html) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2023, 4:30am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/2 "2023-06-18T04:30:58Z")
How are you currently indexing data into Elasticsearch? As you have linked to the JavaScript client I assume you are using a custom JavaScript application. Is this correct? What type ofv data are you indexing? What is the average size ifva document? How much data do you have to index? What is the size and specification of your cluster? ---
**Author:** ![shameel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shameel/32/122398_2.png) [@shameel](https://discuss.elastic.co/u/shameel)\ **Post date:** [June 18, 2023, 5:17am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/3 "2023-06-18T05:17:43Z")
yes im using nodejs client. im working on a personal project with maximmum 5000 json documents ( mostly firewall logs ) being ingested per second to an 8 node cluster which is running in docker overlay network ( actually 4 virtual machines connected via docker overlay ) with 3 masters 4 data nodes and one coordinate nodes. I'm currently using 8 es clients to bulk ingest data. this 5k mentioned is altogether from these 8 clients per second, a single bulk request may contain data destined to multiple indices having 4 shards and one replica. Due to high CPU and Network load on my cluster its very unstable (data nodes being disconnected frequently). im just trying to find the max limit where my cluster can hold? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2023, 5:53am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/4 "2023-06-18T05:53:28Z")
> [@shameel](#): > > Due to high CPU and Network load on my cluster its very unstable (data nodes being disconnected frequently). im just trying to find the max limit where my cluster can hold? It sounds like you may have hit the limitsof what your Elasticsearch cluster can handle. If the cluster is struggling I see no benefit in increasing load or concurrency from the client side. > [@shameel](#): > > a single bulk request may contain data destined to multiple indices having 4 shards and one replica. How many indices and shards are you concurrently indexing into? Are you using time-based indices? > [@shameel](#): > > an 8 node cluster which is running in docker overlay network ( actually 4 virtual machines connected via docker overlay ) with 3 masters 4 data nodes and one coordinate nodes. What is the specification of the VMs and how much resources are assigned to the different nodes? Which nodes are you sending bulk requests to? What type of storage are you using for Elasticsearch? ---
**Author:** ![shameel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shameel/32/122398_2.png) [@shameel](https://discuss.elastic.co/u/shameel)\ **Post date:** [June 18, 2023, 4:07pm UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/5 "2023-06-18T16:07:08Z")
At a time my javascript clients are ingesting data to almost 15 indices. Im not using any in build ilm or time based indices. My VM configurations and the nodes running are PC1 ( 6 Core/20 GB RAM/1TB HDD) - master-a (HEAP 2GB) - data-1 (HEAP 16 GB) PC2 ( 6 Core/20 GB RAM/1TB HDD) - master-b (HEAP 2GB) - data-2 (HEAP 16 GB) PC3 ( 4 Core/15 GB RAM/1TB HDD) - master-c (HEAP 2GB) - data-3 (HEAP 10 GB) PC4 ( 4 Core/15 GB RAM/1TB HDD) - coordinate (HEAP 4GB) - data-4 (HEAP 10 GB) I'm ingesting data to only data-1 and data-2 . data-3 and data-4 are not in use as of now (these I'm planning to keep a different set of data in future - mostly non logs) I restricted index allocation only to 1st two data nodes using "index.routing.allocation.require.box\_type". ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 18, 2023, 4:50pm UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/6 "2023-06-18T16:50:06Z")
> [@shameel](#): > > At a time my javascript clients are ingesting data to almost 15 indices. If bulk requests can target all 15 indices with up to 60 primary shards, you are going to end up with a lot of small writes, which can be inefficient, especially if you are using slow HDD storage. Given that you are using HDDs, [which are not ideal for high indexing loads](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/tune-for-indexing-speed.html#_use_faster_hardware), I would recommend you check iowait and disk utilisation to see wthether this is a bottleneck, e.g. using `iostat -x`. I would recommend upgrading to SSDs and ensure each bulk requests target a minimum of different indiices. > [@shameel](#): > > PC1 ( 6 Core/20 GB RAM/1TB HDD) > > - master-a (HEAP 2GB) > - data-1 (HEAP 16 GB) > > PC2 ( 6 Core/20 GB RAM/1TB HDD) > > - master-b (HEAP 2GB) > - data-2 (HEAP 16 GB) When running Elasticsearch the recommendation is to not set the heap above 50% of the RAM available to the node. Your settings seem a lot higher than this, which is not good. If the master node on PC1 is allocated 4GB of RAM, it should have the heap set to 2GB while the data node. should have a heap of 8GB as it has 16GB RAM allocated to it. Th same applied to the other nodes. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 20, 2023, 4:23am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/7 "2023-06-20T04:23:26Z")
Welcome to our community! 😃 > [@shameel](#): > > i Im using Elasticsearch v7.5.0 Please note that version is [EOL](https://www.elastic.co/support/eol) and no longer supported, you should be looking to upgrade as a matter of urgency. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 18, 2023, 4:24am UTC](https://discuss.elastic.co/t/multiple-child-inastances-of-a-single-client-or-multiple-clients-which-is-better-for-bulk-indexing-in-large-rates/336293/8 "2023-07-18T04:24:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.