# Garbage collection issue **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2019, 9:01am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126 "2019-06-04T09:01:50Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 4, 2019, 9:01am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/1 "2019-06-04T09:01:50Z")
I am trying to bulk index data of size **6GB** , but garbage collection is taking long time so process is getting killed. ``` cluster.name: Elastic-Demo node.name: es-node-1 node.master: true path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 178.90.91.223 hosts.resolve_timeout: 15 http.port: 9200 discovery.seed_hosts: ["178.90.91.223","178.90.91.146","178.90.91.90"] cluster.initial_master_nodes: ["178.90.91.223"] ``` Heap size is 1g. the logs ``` [2019-06-04T13:28:31,570][WARN][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9751] overhead, spent [1.2s] collecting in the last [1.3s] [2019-06-04T13:28:37,024][WARN][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][young][9752][913] duration [5.3s], collections [1]/[5.4s], total [5.3s]/[29.6m], memory [2.7gb]->[2.9gb]/[3.9gb], all_pools {[young] [3.6mb]->[86.8mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [2.7gb]->[2.8gb]/[3.8gb]} [2019-06-04T13:28:37,025][WARN][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9752] overhead, spent [5.3s] collecting in the last [5.4s] [2019-06-04T13:28:39,261][WARN][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][young][9754][914] duration [1.2s], collections [1]/[1.2s], total [1.2s]/[29.6m], memory [2.9gb]->[2.9gb]/[3.9gb], all_pools {[young] [100.5mb]->[1.4mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [2.8gb]->[2.9gb]/[3.8gb]} [2019-06-04T13:28:39,261][WARN][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9754] overhead, spent [1.2s] collecting in the last [1.2s] [2019-06-04T13:28:40,262][INFO][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9755] overhead, spent [289ms] collecting in the last [1s] ``` How do i solve this Thanks. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 4, 2019, 9:06am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/2 "2019-06-04T09:06:44Z")
What's your heap size? What version are you on? What sort of data is it? How many shards+indices and how big are they? How are you sending it to Elasticsearch? ---
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 4, 2019, 9:19am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/3 "2019-06-04T09:19:08Z")
heap size: 1g es version : 7.1.1 the data is basically an LDIF file which i convert to json and it has large number of fields probably hundred.(6GB file) - ``` index shard prirep state docs store ip node ``` - ``` .kibana_1 0 r STARTED 8 49kb 178.90.91.90 es-node-3 ``` - ``` .kibana_1 0 p STARTED 8 49kb 178.90.91.146 es-node-2 ``` - ``` .kibana_task_manager 0 p STARTED 2 12.8kb 178.90.91.146 es-node-2 ``` - ``` .kibana_task_manager 0 r STARTED 2 12.8kb 178.90.91.223 es-node-1 ``` - ``` ldap-listener-stat 2 p STARTED 5290 1.2mb 178.90.91.90 es-node-3 ``` - ``` ldap-listener-stat 2 r STARTED 5290 1.3mb 178.90.91.146 es-node-2 ``` - ``` ldap-listener-stat 1 p STARTED 5217 1.2mb 178.90.91.146 es-node-2 ``` - ``` ldap-listener-stat 1 r STARTED 5217 1.3mb 178.90.91.223 es-node-1 ``` - ``` ldap-listener-stat 0 p STARTED 5155 1.2mb 178.90.91.90 es-node-3 ``` - ``` ldap-listener-stat 0 r STARTED 5155 1.2mb 178.90.91.223 es-node-1 ``` i have no other indices, the index that i was trying to bulk index into has been deleted. I have a python script which takes LDIF file converts to json and index using Parallel\_bulk. ---
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 6, 2019, 4:20am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/4 "2019-06-06T04:20:36Z")
Hi @warkolm, Can you suggest anything. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 6, 2019, 6:14am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/5 "2019-06-06T06:14:34Z")
How large are the documents? What bulk size are you using? ---
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 6, 2019, 6:26am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/6 "2019-06-06T06:26:20Z")
``` if __name__ == ' __main__': pb = parallel_bulk(es, insert_data("user.ldif"),chunk_size=10000, thread_count=8, queue_size=16,request_timeout=10000) deque(pb, maxlen=0) ``` This is the main function and has info you needed ---
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 6, 2019, 6:27am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/7 "2019-06-06T06:27:35Z")
The file is of 6GB, so the insert data function converts it to json . Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 6, 2019, 6:40am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/8 "2019-06-06T06:40:08Z")
Right, but one big document or multiple? ---
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 6, 2019, 6:45am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/9 "2019-06-06T06:45:23Z")
multiple documents ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 6, 2019, 7:23am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/10 "2019-06-06T07:23:00Z")
Try using smaller bulk sizes and see if that helps. Ever larger bulk sizes foes not necessarily give better performance. ---
**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\ **Post date:** [June 6, 2019, 8:41am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/11 "2019-06-06T08:41:26Z")
I tried reducing chunk size to 1000, and it still doesn't work. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 4, 2019, 8:56am UTC](https://discuss.elastic.co/t/garbage-collection-issue/184126/12 "2019-07-04T08:56:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.