Garbage collection issue

I am trying to bulk index data of size 6GB , but garbage collection is taking long time so process is getting killed. Elastic-Demo es-node-1
node.master: true /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
hosts.resolve_timeout: 15
http.port: 9200
discovery.seed_hosts: ["","",""]
cluster.initial_master_nodes: [""]

Heap size is 1g.

the logs

[2019-06-04T13:28:31,570][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9751] overhead, spent [1.2s] collecting in the last [1.3s]
[2019-06-04T13:28:37,024][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][young][9752][913] duration [5.3s], collections [1]/[5.4s], total [5.3s]/[29.6m], memory [2.7gb]->[2.9gb]/[3.9gb], all_pools {[young] [3.6mb]->[86.8mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [2.7gb]->[2.8gb]/[3.8gb]}
[2019-06-04T13:28:37,025][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9752] overhead, spent [5.3s] collecting in the last [5.4s]
[2019-06-04T13:28:39,261][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][young][9754][914] duration [1.2s], collections [1]/[1.2s], total [1.2s]/[29.6m], memory [2.9gb]->[2.9gb]/[3.9gb], all_pools {[young] [100.5mb]->[1.4mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [2.8gb]->[2.9gb]/[3.8gb]}
[2019-06-04T13:28:39,261][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9754] overhead, spent [1.2s] collecting in the last [1.2s]
[2019-06-04T13:28:40,262][INFO ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9755] overhead, spent [289ms] collecting in the last [1s]

How do i solve this


What's your heap size? What version are you on? What sort of data is it? How many shards+indices and how big are they? How are you sending it to Elasticsearch?

heap size: 1g
es version : 7.1.1
the data is basically an LDIF file which i convert to json and it has large number of fields probably hundred.(6GB file)

  • index                shard prirep state   docs  store ip            node
  • .kibana_1            0     r      STARTED    8   49kb  es-node-3
  • .kibana_1            0     p      STARTED    8   49kb es-node-2
  • .kibana_task_manager 0     p      STARTED    2 12.8kb es-node-2
  • .kibana_task_manager 0     r      STARTED    2 12.8kb es-node-1
  • ldap-listener-stat   2     p      STARTED 5290  1.2mb  es-node-3
  • ldap-listener-stat   2     r      STARTED 5290  1.3mb es-node-2
  • ldap-listener-stat   1     p      STARTED 5217  1.2mb es-node-2
  • ldap-listener-stat   1     r      STARTED 5217  1.3mb es-node-1
  • ldap-listener-stat   0     p      STARTED 5155  1.2mb  es-node-3
  • ldap-listener-stat   0     r      STARTED 5155  1.2mb es-node-1

i have no other indices, the index that i was trying to bulk index into has been deleted.

I have a python script which takes LDIF file converts to json and index using Parallel_bulk.

Hi @warkolm,
Can you suggest anything.

How large are the documents? What bulk size are you using?

if __name__ == '__main__':
    pb = parallel_bulk(es, insert_data("user.ldif"),chunk_size=10000, thread_count=8, queue_size=16,request_timeout=10000)
    deque(pb, maxlen=0)    

This is the main function and has info you needed

The file is of 6GB, so the insert data function converts it to json .


Right, but one big document or multiple?

multiple documents

Try using smaller bulk sizes and see if that helps. Ever larger bulk sizes foes not necessarily give better performance.

I tried reducing chunk size to 1000, and it still doesn't work.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.