# Slowness in ES bulk inserts **URL:** \ **Category:** Elasticsearch\ **Created:** [November 27, 2020, 2:00pm UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886 "2020-11-27T14:00:48Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Chirag\_Patwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chirag_patwa/32/79813_2.png) [@Chirag\_Patwa](https://discuss.elastic.co/u/Chirag_Patwa)\ **Post date:** [November 27, 2020, 2:00pm UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/1 "2020-11-27T14:00:48Z")
We are running behind the schedule to achieve something and we need help. Please take a look and suggest something. Issue: Slowness on bulk Indexing. For 10000 records its taking 27560 ms For 500 records its taking about 4448 ms - We want to index about 100,000 documents and its taking 372023 ms which is too much. - We tried different bulk size from 500, 20000,100000 but not able to achieve desired result. - We tried with JestClient, RestHighLevelClient, BulkProcessor but nothing is helping System Configurations: - OS: Linux (debian 9.12) - Standard DS2 v2 (2 vcpus, 7 GiB memory) - 3 nodes configured jvm.options: -Xms5g -Xmx5g -XX:+UseG1GC -XX:+UseStringDeduplication -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly elasticsearch.yml ```auto // # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: hydroperformance # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: hydoperfes0 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /opt/bitnami/elasticsearch/data # # Path to log files: # #path.logs: /path/to/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # #http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["hydoperfes0","hydoperfes1","hydoperfes2"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # discovery.zen.minimum_master_nodes: 2 # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true transport.tcp.port: 9300 network.publish_host: 10.0.0.6 discovery.initial_state_timeout: 5m gateway.expected_nodes: 3 indices.memory.index_buffer_size: 30% ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ``` Thanks In advance ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 27, 2020, 2:46pm UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/2 "2020-11-27T14:46:21Z")
> [@Chirag\_Patwa](#): > > -Xms5g > -Xmx5g You should not set this higher than 50% of available RAM, which in this case is 3.5GB. > [@Chirag\_Patwa](#): > > -XX:+UseG1GC > -XX:+UseStringDeduplication > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly I strongly recommend using the default settings and not override these expert level settings. > [@Chirag\_Patwa](#): > > Standard DS2 v2 (2 vcpus, 7 GiB memory) Have you identified what is limiting throughput? Assuming you are not seeing long or frequent GC it is often either CPU usage or disk I/O, so it is important to use fast storage. See [this section in the docs for more information and tips](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/tune-for-indexing-speed.html). Also make sure you are indexing into enough shards to get all the nodes involved. If you index into a single index with the default 1 primary shard only one or two nodes will be doing work. ---
**Author:** ![egalpin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egalpin/32/79772_2.png) [@egalpin](https://discuss.elastic.co/u/egalpin)\ **Post date:** [November 27, 2020, 9:17pm UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/3 "2020-11-27T21:17:10Z")
In the docs linked by @Christian_Dahlqvist it briefly mentions "unsetting" refreshes, but doesn't expand. One can disable refresh entirely by setting `refresh_interval=-1` on an index's settings if you need to perform a 1-time large-scale ingestion event. Docs will be largely unsearchable during until refresh is re-enabled, but it can speed things up for initial data loads. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 28, 2020, 1:18am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/4 "2020-11-28T01:18:29Z")
> [@Chirag\_Patwa](#): > > Standard DS2 v2 (2 vcpus, 7 GiB memory) Are you on Azure, correct? Is your storage using SSD? Have you tested to see if how many IOPS it is getting? I had a problem a couple of years ago on Azure that even using the Premium SSD based storage the IOPS were pretty low, only solved after opening a ticket to support to check the hardware behind it, which was failing. ---
**Author:** ![Chirag\_Patwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chirag_patwa/32/79813_2.png) [@Chirag\_Patwa](https://discuss.elastic.co/u/Chirag_Patwa)\ **Post date:** [November 28, 2020, 3:46am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/5 "2020-11-28T03:46:08Z")
I tried disabling it from settings API on index, that didn't help in our case ---
**Author:** ![Chirag\_Patwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chirag_patwa/32/79813_2.png) [@Chirag\_Patwa](https://discuss.elastic.co/u/Chirag_Patwa)\ **Post date:** [November 28, 2020, 3:48am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/6 "2020-11-28T03:48:06Z")
Yes. its on Azure portal. I will check on SSD and IOPS. I will post my findings ---
**Author:** ![Chirag\_Patwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chirag_patwa/32/79813_2.png) [@Chirag\_Patwa](https://discuss.elastic.co/u/Chirag_Patwa)\ **Post date:** [November 28, 2020, 3:49am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/7 "2020-11-28T03:49:34Z")
I will try setting 50% RAM and reverting other configurations to see how it behaves. ---
**Author:** ![Chirag\_Patwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chirag_patwa/32/79813_2.png) [@Chirag\_Patwa](https://discuss.elastic.co/u/Chirag_Patwa)\ **Post date:** [November 30, 2020, 9:53am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/8 "2020-11-30T09:53:19Z")
@Christian_Dahlqvist, We tried following as you suggested > [@Christian\_Dahlqvist](#): > > You should not set this higher than 50% of available RAM, which in this case is 3.5GB. Did this change didn't get any improvement for indexing about 500 document performance was the same its was taking average 3seconds. > [@Christian\_Dahlqvist](#): > > I strongly recommend using the default settings and not override these expert level settings. Reverted this still no gain. > [@Christian\_Dahlqvist](#): > > Have you identified what is limiting throughput? Assuming you are not seeing long or frequent GC it is often either CPU usage or disk I/O, so it is important to use fast storage. See [this section in the docs for more information and tips](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/tune-for-indexing-speed.html). Also make sure you are indexing into enough shards to get all the nodes involved. If you index into a single index with the default 1 primary shard only one or two nodes will be doing work. We have by 5 shards configured. Also, followed the link and did following: - Changed setting for a index , set refresh\_interval to 90s and did indexing but no performance gain. - Used multiple worker thread to index, but each individual thread was taking average 3 seconds for 500 documents. - We just have 1 replicas. Just for your information we have unmanaged standard HDD, do you think that might be slowing us down? ---
**Author:** ![Chirag\_Patwa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chirag_patwa/32/79813_2.png) [@Chirag\_Patwa](https://discuss.elastic.co/u/Chirag_Patwa)\ **Post date:** [November 30, 2020, 9:55am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/9 "2020-11-30T09:55:54Z")
We are using unmanaged Standard HDD. Do you think that will not give us milliseconds response time for indexing 500 documents? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2020, 10:04am UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/10 "2020-11-30T10:04:28Z")
Storage performance is often the limiting factor. I would recommend you try with premium storage and see what impact that has. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 30, 2020, 2:32pm UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/11 "2020-11-30T14:32:50Z")
Probably not, those unmanaged Standard HDD are pretty slow for elasticsearch. You should try the Premium tier, even the unmanaged premium tier is better than the HDD, they offer better speed, but they are also more expensive. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 28, 2020, 2:32pm UTC](https://discuss.elastic.co/t/slowness-in-es-bulk-inserts/256886/12 "2020-12-28T14:32:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.